# Ethernet over a radio mesh ## The idea Small radio devices, e.g. IEEE 802.15.4, sub-GHz or LoRa boards, should run Mongoose as if they were on an ordinary Ethernet LAN: get an IP address over DHCP, answer HTTP, talk MQTT. No 6LoWPAN, no Thread, no special stack. To do that, the radio interface looks like Ethernet to Mongoose (`MG_TCPIP_L2_ETH`). Its driver splits each Ethernet frame into radio frames, sends them over the air, and the receiver puts them back together. One or more bridges connect the radio network to the wired LAN: a bridge has an Ethernet port and a radio. A node in range of a bridge talks to it directly; a node out of range reaches a bridge through other nodes, which forward its frames. That makes the radio network a mesh. With several bridges, each node uses one of them, e.g. the nearest; the mesh covers a bigger area, and keeps working if a bridge goes down. A PC on the LAN sees the radio nodes as regular hosts on the same LAN. ``` +---- bridge 1 ~~~ node ~~~ node | ~ PC ---- LAN node | ~ +---- bridge 2 ~~~ node ~~~ node ~~~ node ``` `~~~` and `~` are radio hops. Nodes far from a bridge send their frames through nearer nodes. A node in the middle can reach either bridge. ## Two parts plug in - `struct mg_radio`, a radio standard: how big a radio frame is, how addresses are written into it. `mg_radio_802154` is the first one. - `struct mg_mesh`, a mesh strategy: how nodes find a bridge, and where each frame goes next. `mg_mesh_star` is the first and simplest one: every node talks to a bridge directly, one hop. Everything else, splitting and reassembling frames, sending again when a frame is lost, is common code that works the same for every radio and every mesh. The goal is to swap radios and mesh strategies without touching anything else. ## Why a simulator Real boards are slow to test with, and you can't put 500 of them on a desk. The simulator runs a whole network in one process: every node is a real Mongoose instance, only the radio and the air between nodes are emulated. That lets us try mesh strategies with many nodes, and measure how long it takes nodes to join, get an address, and answer a request. ## Virtual time The simulator has its own clock, `simulator::now`, in microseconds, and Mongoose reads it through `mg_millis()`. Nothing waits for real time: the simulator moves the clock forward itself, 250 us per step, and polls every station once per step. A minute of network traffic takes about a second to simulate. `-t SEC` sets how long to run, 10 minutes by default. The step is a trade-off. Each step polls every station's Mongoose, and even an idle poll costs something, so a smaller step means a slower run. A bigger step makes frames wait longer in queues, see below. Measured with 121 nodes and 10 simulated minutes: | Step | Request time, mean | Run time | |------|--------------------|----------| | 1 ms | 6.1 ms | 0.9 s | | 250 us | 1.5 ms | 3.0 s | | 100 us | 0.6 ms | 7.2 s | | 10 us | 0.06 ms | 72 s | ## The setup ``` DHCP server ---------+ 10.0.0.1 | 02:01:00:00:00:00 +-- LAN ---- bridge ~~~ nodes | test station --------+ 10.0.0.2 02:02:00:00:00:00 ``` The LAN is assumed infinitely fast, see "How frames move on the LAN" below. Two stations are on it: - The DHCP server, 10.0.0.1, MAC 02:01:00:00:00:00, gives each node an address made from its MAC: 02:00:00:X:Y:Z gets 10.X+1.Y.Z, so node addresses never clash with the LAN stations' 10.0.0.x. - The test station, 10.0.0.2, MAC 02:02:00:00:00:00, is a Mongoose instance. It waits for the mesh to settle, 60 seconds by default (`-w SEC`), then sends `GET /` to each node in turn, one at a time, at the node's own IP address. A node with no address, or one that doesn't answer within 10 seconds, counts as failed. At the end it prints how many requests succeeded and how long they took. Bridges are on the LAN too, and get every frame, as if in promiscuous mode: a frame for one of their radio nodes isn't addressed to the bridge itself. A bridge forwards every frame it gets from the LAN to the radio, and every frame from the radio to the LAN, with no filtering yet. Bridges and radio nodes get MACs 02:00:00:xx:xx:xx, numbered from 1 in command-line order: 02:00:00:00:00:01, 02:00:00:00:00:02, and so on. ### How frames move on the LAN Every LAN station, the DHCP server, the test station and each bridge, has a LAN interface, `lif`: a `struct mg_tcpip_if` with a receive queue. All of them use the same driver, `s_lan_driver`. There are no cables and no switch: - **Sending (tx).** A station sends a frame with its driver's `tx`, which is `lan_tx()`. It looks at the destination MAC and copies the frame into the receive queues of the stations it is for: - a broadcast or multicast goes to every station but the sender; - a unicast goes to the station with that MAC; - bridges get every frame, as a real bridge in promiscuous mode would: a frame for a radio node is addressed to the node, not to its bridge. That's all `tx` does. Nobody's code runs on the receiving side yet. - **Receiving (rx).** Each station takes the frames from its own queue when the main loop polls it: - the test station runs Mongoose, which reads the queue by itself, as it would with a real Ethernet driver that queues frames from an interrupt; - the DHCP server reads it in `dhcp_poll()`, and replies by sending through its own `lif`; - bridges read it in `bridges_poll()`, and forward it to the radio. So a frame waits in a queue until its receiver's next poll, at most one step of virtual time, 250 us. Ethernet is hundreds of times faster than the radio, so that doesn't matter for what we measure. And since sending never calls the receiver's code, a reply is never sent from inside another station's send. For example, a node asking for an address: 1. The node's DHCP DISCOVER crosses the radio to a bridge, which sends it out of its `lif`. It is a broadcast, so `lan_tx()` puts it into the queues of the DHCP server, the test station and the other bridges. 2. On its next poll, the DHCP server reads it and sends an OFFER, also a broadcast, which lands in the test station's and the bridges' queues. 3. On their next poll, the bridges read the OFFER and send it over the radio to the node. The test station's Mongoose drops it: the test station has a fixed address and no DHCP client. ## Files | File | What | |------|------| | `l2_mesh.h`, `l2_mesh.c` | `struct mg_radio`, `struct mg_mesh`, common code | | `l2_802154.h`, `l2_802154.c` | `mg_radio_802154`, IEEE 802.15.4 | | `star.h`, `star.c` | `mg_mesh_star`, the star mesh | | `simulator.h`, `simulator.c` | The simulator | | `Makefile` | Builds and runs the simulator | ## Where we are It works end to end: nodes get addresses over DHCP, and the test station's requests reach them through the bridges. But the radio is ideal for now: - The radio driver works like the LAN's: it puts whole Ethernet frames straight into receive queues, all of them for a broadcast, else the one with the destination MAC, and bridges get every frame. No range, no loss, no airtime. - Bridges forward everything, with no filtering. - So use one bridge for now. With two, a broadcast from the LAN goes to the radio through both, each hears the other's copy and sends it back to the LAN, and that never ends. Deciding who forwards what, without loops, is the job of `mg_mesh`. `mg_radio` and `mg_mesh` are still stubs. Next steps, one at a time: the air (range, loss, airtime), fragmenting frames into radio frames with `mg_radio`, choosing a bridge and forwarding through nodes with `mg_mesh`, and filtering at the bridges. ## Running ```sh make # 100x100 m, a bridge, 11x11 nodes make X=200 Y=50 ROWS=2 COLS=8 # Another area and grid ``` `make` puts a bridge in the middle of the area, and a node at every corner of a `ROWS` x `COLS` grid over it. The simulator itself takes: ``` ./mesh_sim -area X,Y,Z -bridge X,Y,Z -node X,Y,Z ... -t SEC -w SEC ``` Positions are in metres; `X,Y` means `Z` is 0. Repeat `-bridge` and `-node` for more.