Wednesday, January 18, 2017

Testing Mesh Extender 2.0 Build Process with Farouk from INRIA

This week Farouk from INRIA in Lille, France has been visiting, as we discuss the potential for collaboration.  It looks like what we are doing with Serval and the Mesh Extenders at the moment will be a good fit for the needs of a project that they have at the moment.  As a result, we have spent a couple of days making sure that Farouk can replicate our build process, which also gives us confidence for others building firmware images as well, and transferring as much of the necessary know-how as possible before he heads back to France, and then in a just a couple more days, that I head back to Australia.

In this process, we are also trying to document all of the outstanding software and hardware issues for the Mesh Extenders at, so that we can better coordinate our activities, and also hopefully make it easier for others to contribute or replicate our work.

Tuesday, January 17, 2017

Testing the Super-Capacitor

We have included a 1F 5.5V super-capacitor on the new Mesh Extender board.

The idea of this is so that when power is lost, the unit has a couple of seconds to stop writing to the microSD card, and perhaps tidy things up to ensure a clean boot next time.

However, we haven't managed to get it to work properly yet.

The boards have a 20 Ohm resister in-line with the super-cap, so that there is no unmanaged in-rush current when charging the capacitor from empty.

The capacitor takes a few minutes to fully charge to about 4.8V (5V - the voltage of the zener diode that forms part of the power loss detection circuit).

At this point, it should be possible to then remove the power source, and have the unit operate powered solely by the super-capacitor, until its voltage level drops below about 3.3V.

GPIO24 is the line that reads our input-power-available signal (effectively the input power before passing through a zener diode, after which it charges both the super-capacitor, as well as powering the rest of the board.  To test the system, I configured that GPIO line for input, and had a little shell script that sits in a loop reading from it.

root@OpenWrt:/# cd /sys/class/gpio
root@OpenWrt:/sys/class/gpio# echo 24 > export
root@OpenWrt:/sys/class/gpio# cd gpio24
root@OpenWrt:/sys/devices/virtual/gpio/gpio24# while [ true ]; do cat value; don


It reports 1 when there is an input power supply (other than USB, which is another little problem we need to solve), and 0 when there is no input power supply, but is running on the super-capacitor.  At least that is the theory.

However, with the 20 Ohm resister, the voltage on the low-side is only about 2.2V, much too low to be useful.  Thus I tried putting a 10 Ohm resister in parallel, dropping the combined resistance to about 6.7 Ohms.  That still wasn't enough.

Then I tried bridging the resister by using a multi-meter in current measuring mode after having already charged the capacitor.  This gives effectively 0 Ohms, and provides an upper bound of what is possible with this super-capacitor.  With that configuration, things were more promising, as I saw u-boot try to boot, but as can be seen, as soon as it tried using the DDR RAM, it would reset, as presumably the DDR RAM draws too much power for the super-cap to sustain the supply above 3.3V:


*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1


*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1


*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1


*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1

*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1

*        U-Boot 1.1.4  (Jun  6 2015)        *

AP121 (AR9331) U-Boot for DOMINO v1


As can be seen above, there is no line with 0, before it hits the u-boot reboot loop, indicating that the board does not operate on the super-capacitor,  even for just a few milliseconds.

We'll have to think of an appropriate solution to this problem, but first we need to understand the situation more completely.  

Is the capacitor unable to provide enough current for the DDR RAM?

Is the capacitor too small?

Also, only one of three super-capacitors seemed to work.  The other two might have been damaged at some point in the process, but having 2/3 faulty makes me think that we shouldn't be relying on them.

Monday, January 16, 2017

Mesh Extender 2.0 second revision prototype boards

The second revision of the Mesh Extender 2.0 (ME2.0) PCBs have arrived.  These were manufactured at almost miraculous speed by RF Design ahead of the Christmas break.  They contain fixes for three of the main problems we encountered with the first revision, fixing problems with both the ethernet and serial level-converter interfaces, as well as the bootstrap pull-up and pull-down on the Domino Core Atheros 9k module.  Here is the new PCBs:

Ethernet - both ethernet ports now work nicely.  One is capable of giga-bit, but sometimes it only connects at 100mbit/sec.  We need to understand what is going on there.

Serial port - the serial port now works without problem, since we have switched to a well-known MAXIM part, instead of the TI level converter, that although in principle capable of level-converting serial lines, turns out to be much too glitchy in our situation.  Other people have had similar problems.

The pull-up registers are now correctly set, although we did still have a problem with one of them apparently not being pulled up hard enough, so we have had to install a single resister to ensure that the Ath9k knows to boot from the 16MB flash, instead of from its internal ROM.  This resister can be seen in the photo below:

We now also have both D-SUB connectors on the same side of the PCB, but having double checked with the 3D-printed case prototype, they are both on the wrong side now, so we will have to fix that in the next revision:

We are currently working on building firmware images for this board, so that we can test other functions, which I'll report on in a separate post.

OpenWRT doesn't see packages from custom feed

Just a quick post to document a problem we encountered with adding a feed to an OpenWRT build process for the Serval Mesh Extender, and how we fixed it.

We followed the process in to add a new feed, by:

1. Modifying feeds.conf (copying feeds.conf.default, if required)
2. Running ./scripts/feeds update customfeed, with customfeed replaced with the name of the new feed.
3. Running ./scripts/feeds install -p customfeed,  again with customfeed replaced by the name of the new feed.

However, when we did this, and then ran make menuconfig, our new packages were nowhere to be seen.

We solved this by running the following additional command:

./scripts/feeds install -a customfeed

Wednesday, December 7, 2016

More details about the Mesh Extender prototype PCBs

I have been asked if I can share the current design documents for the Mesh Extender prototypes, which of course I am happy to do.  I have also confirmed with RF Design who are our PCB design partners (they also make the nice RFD900 / RFD868 radios that we use), that we can share the current schematics as well.

Please remember that all of this information is provisional, and subject to change.

Any feedback and suggested improvements welcome.

The images below have all been uploaded at 300dpi, so click on the image to see in higher resolution.

Jumpers and headers

The first the silk-screen, showing the various connectors:

Note that the only LEDs on the unit at the moment are on the ethernet connectors, so take care in working out whether the unit has power or not. We may put an inline LED in the power cable to make it easier to see if it is powered up or not.  Adding LEDs to the injection-moulded housing would increase costs a lot, and make it harder to achieve the IP65/66 rating we are aiming for.

Going through the headers and jumers from left to right, 

1. Internet of Things / Mesh of Things Interfaces

O1 is opto-coupled input 1, O2 is opto-coupled input 2, and R is the relay-driver output.
I.e., there are two inputs and one output that have some sort of electrically protected interface for IoT/Mesh of Things type applications.

We have allowed some extra flexibility,  not just for during the development phase, on where they will actually connect, i.e., to the GPIOs on the radio module (R position), or to GPIOs on the Ath9k in the Domino Core module (D position).  For R, O1 and O2 the centre pin goes to the relay or optocoupler, and the jumper connects this to either (or possibly both if you had some strange need to do so) the radio module or Ath9k GPIO.  The final decision on which position we default to on shipped units will occur as we continue the development process.

2. Options for powering the device

VIN is for selecting the input power type as automotive (i.e. DC supply or lead acid battery connection) or solar panel input. 

Note to self #1: I'll have to check with RFDesign, if it possible for us to make this selection available on one of the external connectors, so that you don't have to open the unit to make this change.  We could, for example, lose one of CTS or RTS on the UART, in order to make this selection possible externally

For battery charging, you need to select the battery chemistry internally:
LiPO Sets the output charge voltage for charging a 2s LiPO4 battery (as there is no support on the charger IC for load balancing of the cells this type is not recommended)
LiFePO Sets the output charge voltage for charging a 2s LiFePO4 battery
Lead Sets the output charge voltage for the charging a 12V lead acid battery
We will likely make this default to LiFePO, or possibly Lead, but of course it can be easily changed by opening the unit and moving a jumper.

3. Connectivity and debugging

USB is the breakout header for the micro USB connector, in case people want to connect some other device to it.

JTAG is the programmer port for the Domino module.

By default, you need to have a cable on the DB25 connector, in order to tie the Radio and Ath9k serial ports together.  This is on purpose, as it allows for use of an external radio connected by a special cable, that can just re-route the UART from the Ath9k to the external radio.  This is how we intend to allow easy connection to HF radios, for example.  If for some reason you want to operate the device without a cable on the DB25, but still want to bridge the UART to the radio interface, there are some pads just above the DB25 connector that can be soldered to short-circuit the UART connection, so that it doesn't require a cable to complete the circuit.


The main thing to note here are the pinouts of the DB25 and DB15 connectors.

These are subject to change! 

All six GPIOs of the RFD900 are exposed on the DB25 connector. These will be used by the radio firmware to check which regulatory domain it is in, and thus, what the allowed frequency and power level is.  It will also share this information (in the form of the 6-bit value) with the Ath9k, where it will be used to tell the firmware updater whether it is allowed to accept 3rd-party firmware or not.

As these values are supplied by the cable, it will be possible for research and development purposes to make your own cable that does not tell the firmware updater to block 3rd-party firmware, even if you received the unit with a cable for a regulatory domain where this is normally enforced.

Anyway, the current pinouts:

1. DB25 - Power and Radio connector

Going from left to right on the connector:

1 - GND
14 - GND
2 - RFD_IO2 (A GPIO from the RFD900/868 radio)
15 - RFD_IO6 (A GPIO from the RFD900/868 radio)
3 - RFD_IO3 (A GPIO from the RFD900/868 radio)
16 - RFD_IO1 (A GPIO from the RFD900/868 radio)
4 - RFD_IO4 (A GPIO from the RFD900/868 radio)
17 - RFD_IO5 (A GPIO from the RFD900/868 radio)
5 - Domino_TX_H (UART TX line on Ath9k module. Should ordinarily be connected to pin 18)
18 - RFD_TX (UART TX line on RFD900 module. Should ordinarily be connected to pin 5) 
6 - RFD_RX (UART RX line for the RFD900 module. Should ordinarily be connected to pin 19)
19 - Domino_RX_H (UART RX line on Ath9k module. Should ordinarily be connected to pin 6)
Note to self #2: Check if UART lines are around the right way here, or if we need to move them, to ensure that UART bridging can be achieved with just solder blobs between adjacent pins.
20 - RFD_CTS
8 - Ath9k CTS
21 - Ath9k RTS
Note to self #3: Do we really need CTS/RTS lines, and are they also paired the right way around?
9 - Domino Ath9k GPIO0
22 - Domino Ath9k GPIO1
10 - Battery negative (-) pole
23 - Battery centre pole for two-cell configurations (Lithium batteries only?)
11  -Battery positive (+) pole
24 - Solar/Automotive power GND
12 - Solar/Automotive power positive (+) pole
25 - GND
13 - +5V

The UART connections to the Ath9k go through a TXB0104 level converter, making them 5V tolerant.

The +5V line (on both the DB25 and DB15 connectors) can be either used to supply a modest amount of power to some external device, or to supply power to the board.  Thus a minimal DB25 power connector would simply connect the +5V and GND, e.g., from a USB cable, and short the appropriate pairs of pins to connect the UART of the radio module to the Ath9k, and optionally short pins to GND or +5V as required to encode the appropriate radio regulatory domain.

2. DB15 Utility / IoT / Mesh of Things connector

Again, from left to right.

Note that on the prototype PCBs this connector is on the opposite side of the PCB to the DB25 connector.  This is not how it will be in the production units.  I expect that we will be able to re-route the connector without changing the pinout, but nothing is certain yet.

15 - +3.3V
7 - Opto-isolated input 1
8 - Domino GPIO14 (no protection)
14 - Domino GPIO13 (no protection)
6 - Opto-isolated input 2
10 - Domino GPIO26 (no protection)
13 - GND
5 - Domino GPIO27 (no protection)
2 - GND
12 - Relay P3 (The centre-pole of the integrated 5V relay)
4 - Domino GPIO17 (no protection)
9 - Domino GPIO15 (no protection)
11 - Relay P5 (The normally disconnected pole of the integrated 5V relay)
3 - Domino GPIO16 (no protection)
1 - +5V

It should be stressed that the GPIO lines on this connector have no electrical protection whatever, and connect directly to the Ath9k on the Core Domino module.  We have exposed them to provide the maximum flexibility possible with this device.

Regarding the relay: 

The relay itself is an IMC03, which is rated up to 240V and loads of up to ~60VA, and could thus potentially directly switch low-power loads, however, we would recommend using it only as a staging relay, to something larger externally, e.g., a low-cost automotive relay if you were planning on switching low-voltage equipment.

When the relay is activated (via the XLNA pin from the Ath9k, or the IO2 line from the RFD radio module), then pins 11 and 12 will be connected, or disconnected otherwise.  This facility can be used to drive higher-voltage relays externally, e.g., to actuate controllers of various kinds.  

I'll post more information about the Mesh Extender design itself as time allows.

In the meantime, it would be great if someone were interested in starting creating a page for the Serval Mesh Extender 1.0 to the OpenWRT hardware table, so that we can collect and maintain the appropriate information there, and look at creating a relevant target in the OpenWrt DD build system, as I'd greatly prefer if we can include the Mesh Extender in OpenWRT's supported hardware.

Tuesday, December 6, 2016

The new Mesh Extender Prototypes have arrived!

After DHL claiming that they couldn't deliver the parcel to our address here in Germany for "technical reasons," we finally have the two prototypes of the new Mesh Extender design.

As a reminder, this design is required for our AusAID grant to pilot Serval in the Pacific.  The new board is much easier to manufacture, and is designed to fit in a custom-designed housing designed to meet the IP65 or IP66 standard.  This is all being done on a shoe-string budget, in part because hardware always costs more to design than you expect.

Here is the side that accepts an RFD900 or RFD868 packet UHF radio, if it is being optionally included, which will be the normal situation.  The round thing is a 1F super-cap, so that the unit can shut itself down gracefully if the battery/solar power supply fails.  

You can tell it's a prototype by the yellow tape over one of the ethernet connector's pins, where we hadn't thought too hard about where the antenna cables would end up.  Also, the D-Sub 15 is on the opposite side of the PCB to the D-Sub 25 connector, when they should probably be on the same side. Hopefully these will be the worst issues we have to deal with.

You can also see the internal USB connector for holding a memory stick, or something else if you want.  We actually hope to not use the USB port, because of the myriad problems we have had using USB memory sticks on battery-powered Mesh Extenders.  Basically the power consumption is too high, the performance is horrible, and the memory sticks get easily corrupted, or even die, if they lose power at the wrong time.

Below you can see both sides of the PCB.  Here the two internal ethernet jacks can be seen, which face downwards, so that cables can be routed inside the water-proof case.  We won't by default be providing external routing for those cables, as they are not normally needed in a Mesh Extender.  We figure that people will just drill holes in the cases in convenient locations if they need ethernet.  Not ideal, but given our extremely limited budget, it was the best we could manage.

The D-Sub 25 connector has 5V in, as well as 9-26ishV input, which can come from a car or truck plug, or from a solar panel.  That is, you can connect one of these to a suitable naked solar panel, without needing any interface circuitry.  Similarly, there are three pins for connecting a 2-cell LiFePO4, LiPo or sealed lead-acid battery.  The multi-chemistry battery charger is included in the unit.  

Dealing with EU Directive 2015/53/EU and potential unhelpful FCC rules

The D-Sub 25 also has device identification pins that are intended to allow the unit to work out which countries regulations apply, to control allowed transmission frequencies, and if required by FCC rules or EU directives to prevent the installation of third party firmware, that this can be done without any DRM locks. That is, by building a "research and development cable," the device will always be able to accept 3rd-party firmware, but when used with the default cable for a country that requires it, will only accept firmware updates that we have signed.  

We have talked about this solution to mis-regulation in a few different places, and understand that it isn't perfect, however, given our pragmatic requirements, it seems the least-bad option available to us.  We are always interested to hear people's thoughts on how we can improve on this, and of course people are always free to make their own firmware that follows their own policy, instead of this one.

This issue might become more important in the near future, if low-cost wireless routers start coming firmware locked due to these counter-productive regulations.

Getting down to business

So now I need to start qualifying these boards, so that we can confirm to our design and manufacture friends when it is safe to finalise the design, and manufacture the units we need for our pilot in the pacific.

There are a number of steps that we need to cover here, and that I would welcome assistance with, including:

1. Bringing the units up for the first time

2. Making the microSD slots work

The boards have two microSDs, because we don't know which method will work with the Ath9k chipset. One is connected to an SPI bus, which would be the preferred option, but may require some careful control of that bus, as it may have other devices connected. The other is connected via GPIOs to the Core Domino module, which should be fairly easily supportable by the Linux GPIO SPI/SD card kernel driver, although performance is likely to be poor.  We are very keen to get at least one of these options working, so that we can avoid all the horrors of using USB that I have described previously.

3. Building Serval Mesh and Mesh Extender packages for the Core Domino, and testing.

4. Porting our firmware patches for the RFD900 radio to the European version, the RFD868.

5. Testing the solar and battery controller interfaces.

6. Testing the ethernet interfaces.

7. Building and testing power cables, including the region identification and regulatory satisfaction functions.

8. Adding support for the super-cap to cleanly shut the thing down when power is lost, and characterising the super-caps performance. Added bonus: Abort shut down when power is restored during the shutdown process.

9. General integration testing for the whole Mesh Extender software suite

10. Help us develop the remaining functions required for the AusAID Pacific Pilot.

Anyone who is willing and able to contribute to these efforts would be greatly appreciated.

There is the practical limitation that we have only two prototypes, and that they need to stay with me here in Darmstadt, Germany.

That said, if anyone is interested in buying their own prototypes to assist in the process, I would be happy to facilitate this, however we don't have funds to subsidise this process at the moment, and because they are being built as prototypes, we would expect that they will be relatively expensive.  However, if we get enough orders to get a batch made at once, we can look into that.

Related to that, at this stage it would seem reasonable to expect that the prototype boards will be able to fit into the final custom injection-moulded cases that we are getting designed, to allow use of the prototypes outdoors in the future -- although nothing is certain until it happens.

Tuesday, November 22, 2016

Exploring the legal situation of disaster communications networks

I'm currently based in Darmstadt, Germany for a few months with the nice folks in the Secure Mobile Communications laboratory (SEEMOO) at TU-Darmstadt.

One of the main things that we are looking at while I am here, is to better understand the current legal situation facing networks like Serval.

This is important, because the telecommunications laws in most countries were made before such networks were beginning to be widely considered.  As a result, the particular characteristics and requirements of these networks are typically not accommodated in the telecommunications regulations of most countries (or at least the ones we have looked at so far).

So we are hoping over the next month or so to compare the situation here in Germany with Australia, and perhaps a few other countries, to see what road-block might be there, and perhaps to propose how such legislation could be amended to better facilitate them.

As a result I have spent much of the last week reading through the EU Directive 2014/53/EU (in both English and German), as well as draft legislation to implement it in Germany and Austria.

There are some points of concern that we have, particularly that open-source development using wireless routers and SDRs would effectively be outlawed, if the legislation is not carefully worded.  This would be a very unfortunate outcome, not just for the open-source communities who volunteer their efforts and donate the results of their efforts to the common good, but to society as a whole, who would be denied the benefits of these activities (most wireless routers are based on a version one of the open-source projects, very often OpenWRT) because modifying the software on a router to patch a security fix would also be illegal, and router vendors could be required to lock down firmware on the routers to prevent it being patched, updated or replaced in the first place.

This all echoes some of the same problems that are coming up in the USA, with the FCC's proposed rules to require firmware-lockdown on routers.

As we have seen with the confusion there, including TP-LINK being fined for locking down router firmware by the FCC, the matter is not a simple one, and certainly it is one that there still seems to be an understanding gap that we need to help regulators bridge, so that they can be enabled to enact regulations that protect these important rights, while addressing other legitimate social and political concerns.