mongoose/test/mesh/README.md

# Ethernet over a radio mesh

## The idea

Small radio devices, e.g. IEEE 802.15.4, sub-GHz or LoRa boards, should run
Mongoose as if they were on an ordinary Ethernet LAN: get an IP address over
DHCP, answer HTTP, talk MQTT. No 6LoWPAN, no Thread, no special stack.

To do that, the radio interface looks like Ethernet to Mongoose
(`MG_TCPIP_L2_ETH`). Its driver splits each Ethernet frame into radio
frames, sends them over the air, and the receiver puts them back together.
One or more bridges connect the radio network to the wired LAN: a bridge has
an Ethernet port and a radio. A node in range of a bridge talks to it
directly; a node out of range reaches a bridge through other nodes, which
forward its frames. That makes the radio network a mesh. With several
bridges, each node uses one of them, e.g. the nearest; the mesh covers a
bigger area, and keeps working if a bridge goes down. A PC on the LAN sees
the radio nodes as regular hosts on the same LAN.

```
             +---- bridge 1 ~~~ node ~~~ node
             |                    ~
  PC ---- LAN                    node
             |                    ~
             +---- bridge 2 ~~~ node ~~~ node ~~~ node
```

`~~~` and `~` are radio hops. Nodes far from a bridge send their frames
through nearer nodes. A node in the middle can reach either bridge.

## Two parts plug in

- `struct mg_radio`, a radio standard: how big a radio frame is, how
  addresses are written into it. `mg_radio_802154` is the first one.
- `struct mg_mesh`, a mesh strategy: how nodes find a bridge, and where each
  frame goes next. `mg_mesh_star` is the first and simplest one: every node
  talks to a bridge directly, one hop.

Everything else, splitting and reassembling frames, sending again when a
frame is lost, is common code that works the same for every radio and every
mesh. The goal is to swap radios and mesh strategies without touching
anything else.

## Why a simulator

Real boards are slow to test with, and you can't put 500 of them on a desk.
The simulator runs a whole network in one process: every node is a real
Mongoose instance, only the radio and the air between nodes are emulated.
That lets us try mesh strategies with many nodes, and measure how long it
takes nodes to join, get an address, and answer a request.

## Virtual time

The simulator has its own clock, `simulator::now`, in microseconds, and
Mongoose reads it through `mg_millis()`. Nothing waits for real time: the
simulator moves the clock forward itself, 250 us per step, and polls every
station once per step. A minute of network traffic takes about a second to
simulate. `-t SEC` sets how long to run, 10 minutes by default.

The step is a trade-off. Each step polls every station's Mongoose, and even
an idle poll costs something, so a smaller step means a slower run. A
bigger step makes frames wait longer in queues, see below. Measured with
121 nodes and 10 simulated minutes:

| Step | Request time, mean | Run time |
|------|--------------------|----------|
| 1 ms | 6.1 ms | 0.9 s |
| 250 us | 1.5 ms | 3.0 s |
| 100 us | 0.6 ms | 7.2 s |
| 10 us | 0.06 ms | 72 s |

## The setup

```
  DHCP server ---------+
   10.0.0.1            |
   02:01:00:00:00:00   +-- LAN ---- bridge ~~~ nodes
                       |
  test station --------+
   10.0.0.2
   02:02:00:00:00:00
```

The LAN is assumed infinitely fast, see "How frames move on the LAN" below.
Two stations are on it:

- The DHCP server, 10.0.0.1, MAC 02:01:00:00:00:00, gives each node an
  address made from its MAC: 02:00:00:X:Y:Z gets 10.X+1.Y.Z, so node
  addresses never clash with the LAN stations' 10.0.0.x.
- The test station, 10.0.0.2, MAC 02:02:00:00:00:00, is a Mongoose
  instance. It waits for the mesh to settle, 60 seconds by default
  (`-w SEC`), then sends `GET /` to each node in turn, one at a time, at the
  node's own IP address. A node with no address, or one that doesn't answer
  within 10 seconds, counts as failed. At the end it prints how many
  requests succeeded and how long they took.

Bridges are on the LAN too, and get every frame, as if in promiscuous mode:
a frame for one of their radio nodes isn't addressed to the bridge itself.
A bridge forwards every frame it gets from the LAN to the radio, and every
frame from the radio to the LAN, with no filtering yet.

Bridges and radio nodes get MACs 02:00:00:xx:xx:xx, numbered from 1 in
command-line order: 02:00:00:00:00:01, 02:00:00:00:00:02, and so on.

### How frames move on the LAN

Every LAN station, the DHCP server, the test station and each bridge, has a
LAN interface, `lif`: a `struct mg_tcpip_if` with a receive queue. All of
them use the same driver, `s_lan_driver`. There are no cables and no switch:

- **Sending (tx).** A station sends a frame with its driver's `tx`, which is
  `lan_tx()`. It looks at the destination MAC and copies the frame into the
  receive queues of the stations it is for:
  - a broadcast or multicast goes to every station but the sender;
  - a unicast goes to the station with that MAC;
  - bridges get every frame, as a real bridge in promiscuous mode would:
    a frame for a radio node is addressed to the node, not to its bridge.

  That's all `tx` does. Nobody's code runs on the receiving side yet.
- **Receiving (rx).** Each station takes the frames from its own queue when
  the main loop polls it:
  - the test station runs Mongoose, which reads the queue by itself, as it
    would with a real Ethernet driver that queues frames from an interrupt;
  - the DHCP server reads it in `dhcp_poll()`, and replies by sending
    through its own `lif`;
  - bridges read it in `bridges_poll()`, and forward it to the radio.

So a frame waits in a queue until its receiver's next poll, at most one
step of virtual time, 250 us. Ethernet is hundreds of times faster than the radio, so that
doesn't matter for what we measure. And since sending never calls the
receiver's code, a reply is never sent from inside another station's send.

For example, a node asking for an address:

1. The node's DHCP DISCOVER crosses the radio to a bridge, which sends it
   out of its `lif`. It is a broadcast, so `lan_tx()` puts it into the
   queues of the DHCP server, the test station and the other bridges.
2. On its next poll, the DHCP server reads it and sends an OFFER, also a
   broadcast, which lands in the test station's and the bridges' queues.
3. On their next poll, the bridges read the OFFER and send it over the
   radio to the node. The test station's Mongoose drops it: the test station
   has a fixed address and no DHCP client.

## Files

| File | What |
|------|------|
| `l2_mesh.h`, `l2_mesh.c` | `struct mg_radio`, `struct mg_mesh`, common code |
| `l2_802154.h`, `l2_802154.c` | `mg_radio_802154`, IEEE 802.15.4 |
| `star.h`, `star.c` | `mg_mesh_star`, the star mesh |
| `simulator.h`, `simulator.c` | The simulator |
| `Makefile` | Builds and runs the simulator |

## Where we are

It works end to end: nodes get addresses over DHCP, and the test station's
requests reach them through the bridges. But the radio is ideal for now:

- The radio driver works like the LAN's: it puts whole Ethernet frames
  straight into receive queues, all of them for a broadcast, else the one
  with the destination MAC, and bridges get every frame. No range, no
  loss, no airtime.
- Bridges forward everything, with no filtering.
- So use one bridge for now. With two, a broadcast from the LAN goes to the
  radio through both, each hears the other's copy and sends it back to the
  LAN, and that never ends. Deciding who forwards what, without loops, is
  the job of `mg_mesh`.

`mg_radio` and `mg_mesh` are still stubs. Next steps, one at a time: the
air (range, loss, airtime), fragmenting frames into radio frames with
`mg_radio`, choosing a bridge and forwarding through nodes with `mg_mesh`,
and filtering at the bridges.

## Running

```sh
make                          # 100x100 m, a bridge, 11x11 nodes
make X=200 Y=50 ROWS=2 COLS=8 # Another area and grid
```

`make` puts a bridge in the middle of the area, and a node at every corner
of a `ROWS` x `COLS` grid over it. The simulator itself takes:

```
./mesh_sim -area X,Y,Z -bridge X,Y,Z -node X,Y,Z ... -t SEC -w SEC
```

Positions are in metres; `X,Y` means `Z` is 0. Repeat `-bridge` and `-node`
for more.