19 KiB
UDP CPQ Patch Assessment
Scope
This assesses the CPQ patch applied to master, with only this receive-side
change:
if (c->is_bsd && c->recv.len > 0) return true;
It deliberately does not require the bsdudp TX framing, peer filtering, or neighbor-cache work. The rule keeps the first unread nonempty datagram and discards later ones.
What The CPQ Patch Provides
When two nonempty UDP datagrams arrive before a BSD application calls
recv() or recvfrom(), the first remains in c->recv and later packets are
discarded. Therefore the first payload is no longer concatenated with the
second one, and its source address remains in c->rem until it is read.
That is a valid deliberately lossy receive policy. It only addresses this one case. The rest of the odd/incorrect behaviors found when initially writing code in the bsdudp branch are detailed in subsequent paragraphs. The rest of the missing functionality also observed before is NOT described here, only what is claimed to be functional is evaluated; these are somehow enumerated at the end of this document, just before the appendixes
See B.1 for the test evidence.
Rate Limit Of The One-Datagram Policy
The retained datagram is a one-slot loss queue: while the BSD application has
not called recv() or recvfrom(), every later nonempty datagram is dropped.
If the application needs 4 ms to read it, independent traffic has these
approximate collision probabilities:
| Aggregate UDP rate | Probability of another datagram within 4 ms |
|---|---|
| 25 packets/s | 9.5% |
| 50 packets/s | 18.1% |
| 100 packets/s | 33.0% |
| 250 packets/s | 63.2% |
| 1000 packets/s | 98.2% |
This is basic queueing theory under an independent-arrivals assumption. See Appendix A for the model and limits of the estimate.
Baseline Failures
bsd_udp_cpq_fail.c and bsd_udp_cpq_peer.c print the wire-level effects for
short receives, empty datagrams, connected peers, and sendto().
The target runs Mongoose's built-in TCP/IP stack. A bridge connects its packet interface to the LAN because this path cannot use BSD sockets: BSD is the API being tested.
The default runner is self-contained for CI: it starts the peer on the same host (Workstation) and exercises every failure except a mismatched Ethernet destination. To show that L2 failure, run the primary peer on a second machine (Laptop):
Laptop: ./bsd_udp_cpq_peer --remote <Mongoose-IP>
Workstation: CPQ_REMOTE_PEER=<Laptop-IP> ./bsd_cpq_test
Remote mode starts a Workstation helper automatically. The Laptop and helper
have distinct MAC addresses. The helper first redirects a connected-socket
reply, then populates the unconnected sendto() socket with its MAC. The
following sendto() names the Laptop IP, but TRACE_UDP=1 shows the helper's
Ethernet destination. The evidence below was collected with Workstation IP
forwarding disabled, so the Laptop cannot receive a frame addressed to the
helper MAC.
Short recvfrom() turns one datagram into two reads
The CPQ code removes only the copied byte count from c->recv. Reading two
bytes of ABCDE leaves CDE in the buffer, so the next recvfrom() returns
it as if it were a new UDP datagram. UDP must discard the unread tail of a
datagram after a non-peek short receive. Dropping later packets does not repair
this: the tail is already in the retained packet.
See B.2 for the test evidence.
A connected UDP socket accepts traffic from any sender
rx_udp() overwrites c->rem and the return L2 address for every accepted
packet. It does not check that an incoming peer matches the peer selected by
connect(). An unrelated sender can therefore place data in a connected
socket and replace its reply destination. The demonstrator sends intruder
from a third peer: the connected receive succeeds when it should return
EAGAIN, and the subsequent send() goes to the intruder rather than the
connected peer.
See B.3 for the test evidence.
Unconnected send() reports success
bsd_dispatch(BSD_OP_SEND) passes UDP bytes to mg_send() without checking
whether the socket has a peer. A real UDP socket has nowhere to send in this
state and reports EDESTADDRREQ. Here, the API reports that the datagram was
accepted even though it has no valid destination.
See B.4 for the test evidence.
sendto() uses one stale L2 destination
BSD_OP_SENDTO changes the destination IP and UDP port in c->rem, but calls
mg_send() with the one L2 MAC currently stored for the socket. That MAC was
learned when a packet arrived, or while a UDP connect() resolved its peer;
there is no resolution or MAC lookup for each sendto() destination. An
application using sendto() for several peers therefore has one usable L2
destination, not one per peer.
- A single-destination application can appear to work: its only peer is the one that caused the MAC to be learned. This is likely the tested case.
- Sending to another local-LAN host changes the IP address but keeps that first
host's MAC. The Ethernet frame reaches the first host, which drops it
because its IP destination names the second host.
sendto()still reports success. - Several off-link destinations can also appear to work if the stored MAC is the common router MAC. The router is the correct L2 next hop for all of them. This does not make the socket generally correct.
- A socket that has neither received a datagram nor been connected has no
usable destination MAC. Its first proactive
sendto()reports success but cannot be delivered correctly.
Receiving an unrelated datagram can also break a destination that was working.
For example, A sends to this UDP port, so the socket stores A's MAC and a
later sendto(A) works. B then sends any datagram to the same port.
rx_udp() replaces the stored MAC with B's. The next sendto(A) still names
A in the IP header, but addresses the Ethernet frame to B; A no longer gets
it. The same overwrite occurs after UDP connect(), because the CPQ code does
not filter non-peer senders. Remote traffic through the same router can hide
this again because its Ethernet source MAC remains the router's MAC.
The two-host remote test covers both cases: B.5
shows the first sendto() without an L2 address, and
B.6 shows a Laptop IP
sent with the helper MAC. Two sockets on one Linux host are not sufficient for
the latter because they share one NIC MAC. The bridge prints each outgoing UDP
IP destination beside its Ethernet destination MAC, independently confirming
the wrong L2 destination.
(In branch bsdudp, we fixed this by using a framing transporting TX records between the BSD side and the Mongoose side; a correct sendto() must wait
for ARP resolution without losing that destination, so those records/frames are queued at the send buffer and the extraction then can decide whether to start an ARP, thing that is minimized with a small cache-like table holding 4 entries)
See B.5 and B.6 for the test evidence.
sendto() changes the meaning of later send() calls
The same direct update makes an unconnected socket look connected after a
sendto(). A later send() succeeds and uses the previous sendto() target,
although BSD requires EDESTADDRREQ until connect() selects a peer. It is
state corruption from the application's point of view: an operation intended
to name one datagram's destination changes the socket's default destination.
See B.6 for the test evidence.
bind() followed by UDP connect() loses the bound port
mg_connect_resolved() assigns an ephemeral local port unconditionally.
Calling connect() on a UDP socket already bound to a specific port replaces
that port. This is independent of receive buffering; the local identity used
by replies changes after connect().
See B.7 for the test evidence.
sendto() on a connected UDP socket reports success
A connected UDP socket already has a selected peer. sendto() must fail with
EISCONN, but the CPQ implementation accepts it and changes that peer.
See B.8 for the test evidence.
Oversized UDP writes report success after truncation
The CPQ code passes an oversized payload through mg_send(). The built-in UDP
path trims to its payload capacity and transmits only that prefix. BSD callers
are told that the whole datagram was sent. A UDP datagram must be sent whole
or fail; partial transmission is not meaningful.
See B.9 for the test evidence.
Empty datagrams disappear
An empty UDP datagram does not increase c->recv.len. No later recv() can
distinguish it from no packet, so it returns EAGAIN or blocks instead of
returning zero. The proposed condition is also false for an empty retained
datagram, so it cannot represent or protect one.
Yes, it has been agreed that is a known and accepted behavior.
For completeness, see B.10 for the test evidence.
bsdudp Branch Fixes and Additions
These are additional capabilities, not prerequisites for the one-datagram drop policy above:
- RX drops the complete retained datagram after a short non-peek read.
- RX filters packets from non-peer senders on connected UDP sockets.
- TX records carry each UDP destination, so queued
sendto()calls retain their individual targets. - TX triggers and reuses L2 resolution for
sendto()destinations. - Unconnected
send()returnsEDESTADDRREQ; connected UDPsendto()returnsEISCONN. bind()followed by UDPconnect()keeps the selected local port.- BSD writes reject oversized datagrams with
EMSGSIZEbefore queuing them.
Empty UDP receives still require an explicit marker or framing record and are not solved by either the CPQ patch or the current bsdudp simplification.
Appendix A: One-Slot Queue Model
Let D be the time from receiving a datagram until the BSD application reads
it. Let N peers each send at r packets/s, so the aggregate arrival rate is
lambda = N * r. For independent arrivals approximated as a Poisson process,
the probability that at least one later datagram arrives during the unread
window is:
P(collision) = 1 - exp(-lambda * D)
The table above uses D = 4 ms. It is a collision probability, not a precise
packet-loss percentage: the latter also depends on the scheduler and on when
the application calls recv().
For evenly spaced periodic traffic, the aggregate packet spacing must be at
least D to avoid drops. Therefore each of N equal-rate peers needs a period
of at least N * D: with D = 4 ms, 10 peers need at least 40 ms per packet
per peer, while 100 peers need at least 400 ms.
Appendix B: Two-Host Test Evidence
This run used Mongoose at 192.168.69.231, the Laptop at
192.168.69.73, and a Workstation helper at 192.168.69.11. The Laptop MAC
learned for normal control traffic was 3c:0a:f3:8c:ce:ef; the helper MAC was
06:3f:ad:7e:3f:1c. Startup and
DHCP are omitted below.
B.1 CPQ unread-datagram policy
Test. Two peers send one and two before the application reads port
1234.
Expected. The CPQ rule retains one and drops two.
Observed.
Workstation: PASS drop: kept one, dropped two
Laptop: no separate observation
Bridge: no separate observation
Internal reason. rx_udp() returns once a BSD connection has a nonempty
c->recv. It is a one-datagram loss queue, not a UDP receive queue.
B.2 Short recvfrom()
Test. A peer sends one five-byte datagram, ABCDE. The application first
reads two bytes, then reads again.
Expected. The first call returns AB; UDP discards the unread CDE tail.
The second call returns EAGAIN until another datagram arrives.
Observed.
Workstation: FAIL short: expected EAGAIN, got 3 "CDE": tail retained
Laptop: sent ABCDE
Bridge: no separate observation
Internal reason. BSD_OP_RECVFROM removes only the copied byte count from
the stream-shaped c->recv iobuf. It has no record length to discard the rest
of the datagram.
B.3 Connected UDP accepts another peer
Test. The socket connects to the Laptop. The Workstation helper then sends
intruder to that connected socket.
Expected. The helper datagram is ignored. A following send() still goes
to the Laptop.
Observed.
Workstation: FAIL peer: expected EAGAIN, got 8 "intruder": no peer filter
Mongoose: mg_connect_ 7 192.168.69.231:64103 -> 192.168.69.73:14000
Mongoose: mg_connect_ 7 ARP lookup...
Mongoose: rx_arp 7 ARP resolved 192.168.69.73 -> 3c:0a:f3:8c:ce:ef
Workstation: reply: sent 5 bytes; peer reports its destination below
Laptop: FAIL reply: expected control, got none: intruder replaced peer
Helper: FAIL connected peer: helper received reply
Bridge: UDP IP dst 192.168.69.11 -> Ethernet dst 06:3f:ad:7e:3f:1c
Internal reason. mg_connect_resolved() correctly starts an ARP lookup for
the configured Laptop peer and stores its MAC. rx_udp() then accepts every
sender matching the local port. Before the BSD read runs, it overwrites
c->rem and the connection return MAC with the helper's source. The later
send() therefore replies to the helper instead of the peer that was resolved
at connect() time.
B.4 Unconnected send()
Test. Send one byte on a bound UDP socket that has never called
connect() or sendto().
Expected. send() fails with EDESTADDRREQ: there is no peer.
Observed.
Workstation: FAIL send: expected EDESTADDRREQ, got 1: no peer check
Laptop: no packet expected
Bridge: UDP IP dst 0.0.0.0 -> Ethernet dst 00:00:00:00:00:00
Internal reason. BSD_OP_SEND calls mg_send() without checking whether
the UDP socket has a selected peer. The zeroed remote address reaches the L2
transmit path.
B.5 First unconnected sendto()
Test. Call sendto() to the Laptop before this socket has received any
packet or connected to a peer.
Expected. The implementation resolves the Laptop L2 address, waits if needed, then delivers the datagram.
Observed.
Workstation: FAIL sendto: expected delivery, got 4: no resolution
Laptop: FAIL sendto: expected packet, got none
Bridge: UDP IP dst 192.168.69.73 -> Ethernet dst 00:00:00:00:00:00
Internal reason. BSD_OP_SENDTO replaces only c->rem and immediately
calls mg_send(). Unlike the preceding connect() cases, this operation
produces no mg_connect_ ... ARP lookup... or rx_arp ... ARP resolved ...
log for the Laptop destination. It neither performs ARP/L2 resolution nor
retains the destination while resolution would be pending. The zero Ethernet
destination in the bridge trace is the direct result.
B.6 sendto() corrupts later sends and uses a stale MAC
Test. The helper sends a datagram to the unconnected socket, establishing
the helper MAC. The application then calls sendto() to the Laptop and, after
that, calls send() on the same socket.
Expected. sendto() sends to the Laptop MAC without changing the socket's
default peer. The later send() fails with EDESTADDRREQ.
Observed.
Workstation: sendto after RX: got 5
Workstation: FAIL send: expected EDESTADDRREQ, got 4: sendto set peer
Laptop: FAIL stale MAC: expected packet, got none
Bridge: UDP IP dst 192.168.69.73 -> Ethernet dst 3c:0a:f3:8c:ce:ef
Bridge: UDP IP dst 192.168.69.73 -> Ethernet dst 06:3f:ad:7e:3f:1c
The first bridge line is normal Laptop control traffic. The second names the same Laptop IP but uses the helper MAC. The Laptop receives no packet.
Internal reason. The implementation keeps only one mutable c->rem and
one return MAC per Mongoose UDP connection. Receiving the helper datagram
updates that one MAC to 06:3f:ad:7e:3f:1c. sendto() overwrites c->rem,
but does not start an ARP lookup for the Laptop and therefore retains the
helper MAC. There is no mg_connect_ ... ARP lookup... or rx_arp ... ARP resolved ... sequence between this sendto() and the second bridge line. No
per-datagram destination record or neighbor cache exists.
B.7 bind() followed by UDP connect()
Test. Bind local port 1238, then connect to the Laptop and query the
local address.
Expected. The socket remains bound to 1238.
Observed.
Workstation: FAIL bind: expected 1238, got 64102: connect replaced bind
Mongoose: mg_connect_ 5 192.168.69.231:64102 -> 192.168.69.73:14000
Mongoose: mg_connect_ 5 ARP lookup...
Mongoose: rx_arp 5 ARP resolved 192.168.69.73 -> 3c:0a:f3:8c:ce:ef
Laptop: normal ARP/control peer
Internal reason. mg_connect_resolved() assigns an ephemeral local port
without preserving the BSD socket's existing bind state.
B.8 sendto() on a connected UDP socket
Test. After UDP connect(), call sendto() with a different destination.
Expected. Fail with EISCONN; a connected UDP socket uses send().
Observed.
Workstation: FAIL sendto: expected EISCONN, got 1: connected socket accepted it
Laptop: no separate observation
Bridge: no separate observation
Internal reason. BSD_OP_SENDTO has no connected-UDP state check and
blindly replaces the peer address. The connected socket's ARP lookup happened
at connect() time in B.3; this operation does not validate that state or
start resolution for its replacement destination.
B.9 Oversized UDP write
Test. Send a 1600-byte datagram.
Expected. Fail with EMSGSIZE; UDP cannot partially send one datagram.
Observed.
Workstation: FAIL oversize: expected EMSGSIZE, got 1600: later truncated
Laptop: no receiving socket in this run
Bridge: no payload-length assertion in this run
Internal reason. The BSD layer does not reject an over-capacity datagram. The built-in UDP transmit path later limits payload to its available capacity, while the BSD caller has already been told all 1600 bytes were accepted.
This run proves the false-success API result. A receiver-side payload-length assertion would additionally demonstrate the transmitted prefix length.
B.10 Empty datagram
Test. Send a zero-byte UDP datagram, then call recvfrom().
Expected. recvfrom() returns zero and preserves the source address.
Observed.
Workstation: FAIL empty: expected 0, got -1 errno 11: no marker
Laptop: sent a zero-byte datagram
Bridge: no separate observation
Internal reason. An empty payload does not increase c->recv.len; the
stream iobuf cannot distinguish an empty datagram from no pending data.