| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
| |
one (017, 2b-1)
The sending loop was woken by every record queued while it slept: the
owner queues one by one, the loop caught up and slept between them, and
every link took a round on every wake. Four links sent 64k packets of
12 records where threads sent 40k of 20, and ran at 257k sends/s
against 581k. The owner now queues a turn's sends and pushes each link
once; the loop rounds a link only when its socket has answers, it was
pushed, or its time came.
|
| |
|
|
|
|
|
|
| |
OpenSSL 3.5's ClientHello offers ML-KEM with X25519 and takes two
Initials; when the path brought the second first, ngtcp2's server was
asked for a Retry and the harness gave up. The client now offers X25519
alone, as Pure's links agree, and the server lets go of a connection it
may not keep, so the client's next Initial begins another.
|
| |
|
|
|
|
|
|
|
| |
drain.c takes a rate and seconds as quinn's client does (paced messages
stamped with the time they were written, and no more written past the
seconds), and its service writes quinn's line for each stream: the
messages read in each 100 ms and how long they took. 020's run.py runs
it as a third, on port 40005 behind the same bottleneck, and --who
picks whom to run.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
ngtcp2 1.25.0 with OpenSSL 3.5's QUIC TLS (its ossl backend), fetched
and built into build/third by ngtcp2/build.sh; drain.c does in C what
the quinn harness does: one connection, one stream, "pure" and then the
messages each after its u32 length, counted by the service, packets in
GSO bursts. Its stream bytes are a function of their offset, so what
ngtcp2 keeps to send again points into a pattern that never changes.
run.py's drain_ngtcp2 measures both sides' CPU as drain_quic does.
Five rounds each, both sides' CPU a message: 61 B Pure 0.280, ngtcp2
0.270, quinn 0.310 us; 1 KiB 2.62, 3.54, 3.94; 16 KiB 40.2, 54.4, 60.2.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A receiver that started again without its memo could open none of its
senders' packets (their epochs were gone), said nothing, and each sender
waited 2 s and four probes, then offered with no records. Worse, those
packets failed to open, so each cost a token (256 a second, the flood
guard): ten thousand senders came back 256 a second. M2 without a memo
stopped 0.8 s and caught up in 7.7 s.
Now the receiver answers a packet of a run it does not know, in an epoch
past 0, with LOST: sealed in epoch 0 under the nodes' keys, naming the
packet. As QUIC's stateless reset, it is smaller than the packet and
comes out of a bucket of its own (4,096 a second). A sender that has that
packet out begins again as a run begins: epoch 0 with its records at
once, epoch 1 offered. Only the receiver can say LOST, and one of the past
names a packet no longer out; silence alone (an attacker's doing, maybe)
still gets offers only. Such a packet costs no key token: no keys were
made for it, and the secret made is remembered, once a sender.
M2 without a memo (ten thousand devices, twice): no 100 ms step stopped,
caught up in 0.4-0.5 s, p99 5.8-5.9 ms, the most 48-57 ms (was 4.1 and
7.8 s). test-link: a changed packet may get a LOST smaller than itself
and nothing else; half of a flood in epoch 1 got 267 LOSTs, 12 KB for its
6 MB; a record goes under the nodes' keys after an epoch began only after
the receiver forgot, until the next epoch.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
H3 of design 011 stands: with ten thousand devices sending 2,000 records
a second, the node's store, key and memo carried to another address,
nothing stopped for a 100 ms step, the moved node made no X25519, and
every record was applied once. Each device's first record after the move
waits one probe (30-45 ms here), as its first packet finds no way there.
The search found the listener answering from an address other than the
one a packet came to, and a link that looked for its receiver after its
probe went, losing every answer. H1-H4 are all measured now; 61 B's CPU
a delivery is at the edge of the criterion, to be looked at before step
5 (adoption and a specification).
|
| |
|
|
|
|
|
|
|
|
| |
run.py --restart --move runs in namespaces of its own (unshare -rmn: a
network and an /etc/hosts): the devices send to receiver.test, which
names 127.0.0.2; at --kill-at the node is killed, its store, key and
memo copied elsewhere, 127.0.0.2 made unreachable, the name pointed at
127.0.0.3, and the copy started there. The load generator's devices
look for the node again as link/link.c does: after three probes in a
row unanswered, or a send the system said had no way there.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Forward secrecy, 098's first task left: keys that leak later open only
epoch 0. M2 after the epoch began and after a restart is as before it
(p50 0.4 ms, p99 0.8 ms, the same CPU, no X25519 on restart). It costs a
p50 of 0.7-1.0 ms more while ten thousand devices first meet and for
about 40 s of accepts after, and a receiver that lost its memo now
stalls 0.8 s and catches up in 7.7 s, not 0.2-0.7: it lost the epochs,
and each sender opens one again. The search also found a packet more a
device from each of M6's path probe and the security checks' challenge,
and a load generator whose own pass over its devices was the floor of
the latency it measured. 098's note that epochs narrow KCI to the first
epoch was wrong; corrected.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
nothing past
A sender offers the public half of a pair made for the next epoch
(OFFER) until the receiver accepts with a half of its own (ACCEPT); their
X25519 and the epoch's secret make the next one's, and the private
halves go. The header's low four bits say a packet's epoch; epoch 0 is
the nodes' static secret, as the keys were. The first epoch is offered in
the run's first packet (it begins with the first answer when the
receiver is not busy), then one an hour or GiB. One who later has both
nodes' keys opens only epoch 0, and no record went there after an epoch
began: test-link tries every packet sent under the static keys, with
epochs a second long, loss, reordering, restarts and replays.
The receiver writes the epoch it accepted in its memo (a place is 256
bytes now), so it goes on without an X25519 when it starts again; one
that forgot is not heard from, and its senders, after 2 s and four
probes unanswered, offer again in epoch 0 with no records until an epoch
begins. Keys of four epochs are tried (accepted, now, before, zero), as
two may share the four bits.
An accept costs an X25519 on each side, 220 µs on this Pi: one of a
sender with records costs a token (256 a second, 8 at once) and the
next packet offers again when there is none; one of a sender with
nothing else to send waits for none, one a second for it. The X25519 is
made after the turn's answers, never between a packet and its answer.
serve's report and M2's generator count keys, X25519s and epochs.
|
| |
|
|
|
|
|
|
|
|
|
| |
The load generator asked all ten thousand senders for their timers every
5 ms: 83 % of its CPU (47 ns a sender, mostly cache misses), and records
were timed from the start of that pass, so the pass (0.75-0.94 ms, longer
as the sender grew) was the floor of every latency it measured. A device
now keeps its sender's next timer, asked again only when the sender is
touched (the only time it changes), and the p50 of M2 falls from 0.8 to
0.4 ms for every version alike. It also says the p50 and p99 of the
records made in each second, and the keys and X25519s its senders made.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
acknowledged
A link starts at 1,200 bytes; once an RTT is known it sends a PING padded
to 1,452 (DPLPMTUD, RFC 8899), alone and with the don't-fragment bit,
and uses that size once it is acknowledged. A probe lost is no
congestion; it is tried again a second on, three times, then ten minutes
on. GSO is asked for each burst (a control message), not the socket, so
a packet sent alone is never cut. CUBIC counts in the datagram.
On loopback packets fall by 19-27%; the CPU a record falls 13% at 1 KiB
and 7% at 16 KiB; in the simulator of a 50 Mbit/s path with 1% loss the
goodput rises from 6.1 to 7.0 Mbit/s (5%: 2.3 to 3.0).
|
| |
|
|
|
|
|
|
| |
tbf with netem under it lost tbf's queue (netem became it), and the
answers queued behind the data. Now each way has its netem, the way to
the receivers' ports then a tbf whose queue holds a round trip; the way
back does not queue. transport.c takes its listener's port
(TRANSPORT_PORT).
|
| |
|
|
|
|
| |
From the one after the last heard, letting be the ones heard before and
stopping at a gap: the listener's first delivery may begin past it when
packets reorder, which the harness counted as wrong.
|
| |
|
|
|
|
|
|
|
| |
020-lossy-paths/run.py runs each case in an unprivileged namespace: a
50 Mbit/s tbf bottleneck under netem's delay, jitter and loss, both
ways. A link may send without UDP GSO (pure_link_gso), as netem takes a
burst as one packet; quinn the same (QUIC_GSO=0). transport.c and the
quinn harness give the records handed over in each 100 ms, and at a
steady rate the time each took; TCP CUBIC flows run beside them.
|
| |
|
|
|
|
|
|
|
|
| |
links.c was a second implementation of the packet format; each device is
now a PureLinkSender on a socket of its own, keyed with a secret made
outside (pure_link_sender_shared: a key kept elsewhere, or a test's
many). M1 (1k, 10k) and M2 (10k, kill -9) pass with it: no silent step,
caught up in 0.2 s, every order applied once; the slowest store after
the restart is now the sender's 1 s wait for a word of the save
(STORED_WAIT), where the old devices tried again after 200 ms.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
link/link-protocol.h is the protocol with no socket, thread or clock of
its own: packets and time in, packets out. link/link.h carries it over
UDP as before. A sender starts small (8 flights, 16 records, a 4 KiB
ring) so one thread can hold many. The transport's CPU a record is the
same within 2% (five rounds each way, 61 B to 16 KiB).
test-link runs both sides through a network of its own: loss, packets
held back and doubled, failed saves, an idle sweep, and both sides
starting again, every record applied once as sent.
|
| |
|
|
|
|
| |
A sender takes its messages' shape, and a listener asks its host which
input a stream goes to. The core gives the shape (pure_module_shape);
link/ now builds without the core's headers.
|
| |
|
|
|
| |
libpure-link.a holds them; pure-serve and the M5 harness link it instead
of naming its source files.
|
| |
|
|
|
|
|
| |
Between two pure-serve nodes, 16 KiB records drain at 33,600 a second
(23,000 before), the links spending 58 us a record (76), under quinn's 60
while the receiver also stores each one. 1,013 bytes: 406,000 a second
(347,000). Saving sends one by one: 5,000 of 16 KiB in 0.91 s (84.8).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
M5 of design 011. At first a link spent eight times quinn's CPU on a small
record: scans of the backlog, a system call and allocations a record, 19
bytes of framing. After the seven changes before this one, the transport
alone (transport.c; a backlog drained, every record checked) spends per
record 0.30 us for 61 bytes (quinn 0.27: 1.06 to 1.12 times over three
campaigns, at the design's 10 % line), 2.91 us for 1,013 bytes (quinn
3.88) and 45.2 us for 16,013 bytes (quinn 60.9): 25 % less for the larger
ones, and 30 % more records a second. What is left for small records is
per packet: the stream's names and the RECORDS head. quinn's numbers carry
no confirmation of storing and no exactly once.
Between two pure-serve nodes the links spend more per record, woken at
the receiving app's pace. Filling the sender showed the outbox is written
whole at every save, as the numbers heard were (094): the next thing to
change.
run.py gains --transport and --drain; the quinn service reads a drain a
chunk at a time and its client writes 64 KiB at a time.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A record over a piece (1,000 bytes) went in pieces even when it fit the
packet whole: a record of 1,013 bytes went as two RECORD frames, and the
receiver gathered it the slow way (an allocation, pieces, then onto the
line). A record none of whose pieces went yet now goes whole in a RECORDS
frame whenever it fits what is left of the packet; its pieces are marked
sent together. A record too large for a packet, or one resent after
some of its pieces were lost, still goes in pieces, and a receiver takes
a record whole or in pieces alike.
The transport alone takes 2.85 us a record of 1,013 bytes, not 3.18
(quinn: 3.90). transport.c now checks every record delivered: in order,
once each, its bytes as sent. None was wrong in a network namespace with
1 ms of delay, 3 % loss and 10 % reordering, with records of 61, 1,013
and 16,013 bytes.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Every queued record wrote the link's eventfd, a system call each, and woke
the thread to send a packet of a few records. A record queued now wakes
the thread only when it sleeps and nobody woke it yet; one queued while it
sends marks the link, and the thread looks again before it sleeps. With
records queued one at a time as fast as a thread can, 200,000 small ones
went at 400k a second, and now at 730k; queuing one costs 0.43 us, not
1.0.
research/experiments/019-idle-links/transport.c measures the transport
alone: a link and a listener in one process, no app, no store. A backlog
of 200,000 records of 61 bytes drains at 3 to 4 million a second, 0.24 us
a record on each side.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
M2 and M4 of design 011. Ten thousand devices send an order every five
seconds (2,000 a second); the receiver is killed (-9) at about 21 s and
started again at once. With its secrets remembered the node shows no
100 ms step without an order stored, catches up in 0.2 s, spends no more
CPU than before and makes no X25519; 34 orders were sent again, and
every order made was applied once (three rounds; on the SD card the
same, with 0.2 s silent). Without the memo it makes 10,000 X25519, 1.6 s
of CPU. quinn under the same load stops for 1.0 s and catches up after
10.9 s: 10,115 new connections, about 7 s more of the service's CPU and
8 s of the clients'. A first contact takes Pure one round trip and QUIC
two, at 10, 50 and 100 ms of RTT.
run.py --restart and --first-contact make them; the quinn harness gains
a fixed port with a reset key from its key, clients that connect again,
and a drain for M5.
|
|
|
M1 of design 011. A Pure node allows 1,000 to 1,000,000 senders; each
sends one order and, once told it is stored, rests. At rest a link costs
the node 132 bytes (the sender allowed, 73; its number heard, about 60)
and no transport state; with a million the node runs in 139 MB. A quinn
0.11 service holding the same exchange's clients connected at rest costs
38,106 bytes a connection: 289 times as much, past the design's 1/100.
A link at rest takes its next record in 0.56 ms to stored, one X25519 on
the node included; a new QUIC connection takes 1.06 ms and 2.8 times the
receiver's CPU, without storing.
research/experiments/019-idle-links: links.c, many senders written from
serve/link.h's packet format alone (a second implementation), and the
quinn service and clients (quic/, built offline); run.py makes both
measurements.
|
