| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A message from another node was taken only in its input's exact shape
(the link's STREAM shape, fields and order alike; over HTTP, every
field). So a board whose nodes are upgraded one after the other stopped
both ways until both were, and an output and its input whose fields
differ only in order were refused for good (docs/design/014).
An input now takes a message when it has every field the sender sends,
by name, of the same type or one that holds its values (a u32 in a
number, text in bytes); a field only the input has is 0 or empty. One
that sends a field the input lacks is refused: taking it would drop the
value, and with its number heard the value would never come again (two
boards apart for good). So add fields to receivers first and take them
from senders first, and nothing stops or is lost.
The link carries a sender's fields in a new frame, FIELDS (0x0d: names,
shape, u8 count, each {type, name length, name}): after a REFUSED for
another shape the sender puts it in its packets (after OFFER, before
the records, so a receiver that knows no FIELDS reads what it read
before) until the stream is taken, the first time at once. A receiver
seeing another shape waits for the packet's FIELDS before refusing, and
asks its host with them (PureLinkHost.input takes them); REFUSED says 4
when the input lacks a field. Deliveries carry the fields and the host
lays each record out anew (pure_module_takes, pure_module_recast,
pure_module_fields, over pure_send_recast's rule). A stream whose sender
starts again in another shape lets go what it held; the sender sends it
again laid out anew. A refusal already queued for a stream now says the
last reason judged, not the first. Over HTTP a delivery (X-Pure-Source)
may lack fields; an unknown member is still refused.
test-link named: a stream of another shape naming its fields is taken,
200 stored in 47 ms with every delivery naming them; an input that does
not take them refuses (4) and the sender backs off. check.py
server_fields: five outputs, three linked and two over HTTP, fields in
another order taken, an input with a field more taking messages without
it, one lacking a field refusing until it starts again with it, then
taking all. server_upgrade's new receiver declares its fields in another
order. 012 names the frame and the rules.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A sender whose stream the receiver refuses sends one packet of records
a wait, and the refusal is over when that packet is acknowledged with
no REFUSED. When the try itself went to no one (the receiver started
again, as a node upgraded one after the other does, or the path lost
it) nothing acknowledged it: the sender took its epoch for lost, began
another, and then sent nothing more, still waiting on the try, with no
record in flight to probe for. Two nodes of a shared board stayed apart
for good after the second one's upgrade.
A try declared lost, or dropped with the epoch's flights, now lets the
next packet try again. test-link's refused_restarted case: the receiver
refuses a stream, is killed while the next try is due and comes back 8
s later with the input; before, nothing was stored in 120 s, now all
200 are by 18.2 s. Streams refused for good send as few packets as
before (9 in 90 s).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A link's ring and a wire's queue took every saved send of their output
and kept it until delivered: with the receiver down they held the
outbox a second time, 178 B and 313 B more a send (104). Each outlet now
holds a window, --window bytes (4 MiB) and 65,536 sends past what its
receiver has, at least one; the outbox keeps the rest.
The owner remembers each outlet's place in the outbox and looks on from
there after a save and after its transports deliver. Places move back by
the sends that left the outbox since (PureApp.sends_gone), so a trim
before a place skips nothing, and sends queued before are not looked at
again however many wait. pure_app_sent reads a send by its place (the
newest-first visit it replaces is gone); pure_link_held and
pure_wire_held say what a transport holds.
With the receiver down a node now grows by the outbox alone once the
window is full: 219 and 218 B a send over the link and HTTP (399 and
534 B before). check.py server_windows keeps the receivers of a sender's
two outputs, one linked and one over HTTP, down through 300 buys with a
2,000-byte window, then starts them: each applies all 300 once. Without
the refill after deliveries the link stops at 117.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A link's delivery the app had no room for was handed over again at the
next turn: a receiver over its memory budget would refuse the same
records as fast as the owner could apply them. And its sender, seeing
records received but not stored for a second, sent the whole window
again every second.
Now a host says a delivery found it full (PureLinkDone.full). The
receiver keeps the records and hands them over again after a pause, 100
ms doubling to 500 ms, and meanwhile its STORED frames carry flag 2: the
app is full and what came is kept. A sender hearing it starts its
one-second wait for the save again, and sends none of it again. The
spec (012) says both sides; the pause is a local choice that must stay
under the sender's wait.
pure_link_done takes the PureLinkDone itself. test-link's new case keeps
the app full for 5 s: 12 hand-overs meanwhile, taken within 0.5 s of
having room, and no record sent again (21,640 without the flag).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A receiver gave every stream a credit of 1,024 records past its last
stored, however small the records: a link carried at most 1,024 records
a round trip. On a path of 100 ms that is 10,000 records a second of
61 B, a ninth of what 50 Mbit/s carries (the path simulator: 9,953 a
second, 90,814 at 10 ms); and records of 1 KiB, a queue's worth of them
in flight, went a record or two a packet as each STORED let a few more
through (M6: 107,968 packets where 20 ms took 84,106).
A receiver bounds what it holds of a stream (16 MiB) apart, so its
credit is now what fits there at the size its last delivery's records
had: 1 to 65,536 (what a sender takes), 1,024 before any delivery. It
takes records that far past its last stored, as its memory lets it.
Simulator at 100 ms: 56,296 records a second of 61 B. M6 at 100 ms:
45.4 Mbit/s of 1 KiB records in 83,566 packets (44.9 in 107,968). The
window, not the credit, now fills the bottleneck's queue: losses there
where the credit hid them (61 B at 10 ms: 87,130 a second, 1,371 lost).
012 says so.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A receiver's memo kept one run's epoch a peer. A node whose outputs
send to this one sends in a run each, so after the receiver started
again (or let the peer rest) all runs but the last accepted were told
LOST and began their epochs again: records under the static keys for a
round trip, and an X25519 each. Now a run whose peer's place holds
another run still sending keeps its epoch beside, in one of two of
1,024 places its peer and run find (160 KiB at the file's head: the
memo is PLM3, and a PLM2 one is begun again). A run's place beside goes
once its peer's place has it, so an epoch's secrets stay only until the
next one's are.
test-link's three outputs of one node, each in epoch 1, go on after the
receiver starts again: no LOST, no X25519 (before: two LOSTs, an
accept). 012 says so.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A receiver that took no records of a stream (a source the key may not
send as, no input of that name, an input of another shape) said nothing:
its sender heard them acknowledged, never stored, and sent them all
again every second, for ever, and its operator saw nothing. Now the
receiver answers REFUSED (0x0c: the names, and why), and the sender
sends no records for a second, then a packet of them to learn whether
they are taken now, the wait twice as long each time (64 s at most). A
refusal ends when that packet is acknowledged with no REFUSED beside, or
a record is stored. link.c says it once on stderr, as a wire says a
receiver does not take its sends; serve's input says which of the two
the app lacks.
test-link: each of the three is refused seven times in 90 s with nine
packets, and a stream refused until the app has its input goes on then,
all 200 records stored. check.py links a node to an input its receiver
lacks: said once, two packets in 2.5 s.
|
| |
|
|
|
|
|
|
|
|
|
| |
A receiver kept two runs a peer: the sender's, and the one before it
restarted. A node with three outputs to another sends in three runs with
one key, and they pushed one another out: each packet keyed its run
again, and an answer marked for one went to the run that took its place.
Now more places are made beside when the run seen longest ago is still
live (up to 64), and one is reused only once its run is quiet the idle
period. test-link's three outputs of one node key three runs, not seven,
and are all stored in 16 ms, not 1016.
|
| |
|
|
|
|
|
| |
A sender took sends up to 64 MiB, but a receiver holds 16 MiB of a
stream not yet stored, so a send between the two went again forever.
The most a send may be is now PURE_LINK_RECORD_MOST, what a receiver
holds; test-link sends one that large whole and is refused one byte more.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
A RECORDS frame of records in order, with nothing held and no record's
tail waiting, went through hold, on_line and line_append a record: the
same checks each time (heard, credit, the line's end, held, the memory
bound). Now they are made once for the frame and its records are copied
onto the line one after another. The receiver's CPU a 61-byte record in
M5 went from 0.15 to 0.133-0.143 us; both sides together 0.28 us, beside
ngtcp2's 0.27 and quinn's 0.31 in the same rounds (the M5 host here also
compares every record with what was sent). test-link's results are the
same to the byte.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A receiver that started again without its memo could open none of its
senders' packets (their epochs were gone), said nothing, and each sender
waited 2 s and four probes, then offered with no records. Worse, those
packets failed to open, so each cost a token (256 a second, the flood
guard): ten thousand senders came back 256 a second. M2 without a memo
stopped 0.8 s and caught up in 7.7 s.
Now the receiver answers a packet of a run it does not know, in an epoch
past 0, with LOST: sealed in epoch 0 under the nodes' keys, naming the
packet. As QUIC's stateless reset, it is smaller than the packet and
comes out of a bucket of its own (4,096 a second). A sender that has that
packet out begins again as a run begins: epoch 0 with its records at
once, epoch 1 offered. Only the receiver can say LOST, and one of the past
names a packet no longer out; silence alone (an attacker's doing, maybe)
still gets offers only. Such a packet costs no key token: no keys were
made for it, and the secret made is remembered, once a sender.
M2 without a memo (ten thousand devices, twice): no 100 ms step stopped,
caught up in 0.4-0.5 s, p99 5.8-5.9 ms, the most 48-57 ms (was 4.1 and
7.8 s). test-link: a changed packet may get a LOST smaller than itself
and nothing else; half of a flood in epoch 1 got 267 LOSTs, 12 KB for its
6 MB; a record goes under the nodes' keys after an epoch began only after
the receiver forgot, until the next epoch.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
A link that found the path carries 1,452 bytes kept sending them when a
route changed and larger packets were dropped with nothing said (a
black hole): it lost them one after another and stood still (test-link:
8 of 3,000 records in ten minutes). Now, as RFC 8899 §4.3 has it, it goes
back to 1,200 bytes when three larger packets in a row are lost while an
acknowledgment covers only smaller ones, or when two probes in a row go
unanswered, and tries the larger size again a second on (three tries,
then ten minutes). The case completes in 1.2 s of virtual time; the M6
simulator's results are the same to the byte in eight paths of loss,
reordering and a bottleneck.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Profiling both sides in one thread, without sockets (61-byte records):
the rings of records and flights divided for each place, the record a
number names was found three times over, a record in one part was
searched for its part, and the receiver walked the line record by record
to hand it over and again to let it go. The rings are powers of two long
and now masked; a record is found once; a one-part record's part is the
first; a line of a batch or less goes whole without a walk, and a
delivery applied whole lets its bytes go at once. The protocol alone:
0.171 to 0.162 us a record. M5 (the link and listener threads, sockets
too), five rounds against the commit before: 61 B 0.298 to 0.287 us, 1
KiB 2.63-2.90 to 2.48-2.79 us, 16 KiB the same or less; quinn measured
0.31, 3.85 and 60 us in the same rounds. test-link's results are the
same to the byte.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A link resolved its receiver's name once, at its start: a receiver whose
store moved to another machine (another address under the same name)
was never found, and a sender whose own network changed kept a socket
that no longer went anywhere. Now, when three probes in a row go
unanswered, and at each probe after (a second apart at most), the link
looks the name up again and sends from a new socket connected there.
It looks before the probe goes, so the probe's answer comes to the new
socket (looking after it lost every answer, one socket after another).
A sender at rest does not look: only probes unanswered count.
make check moves a receiver (H3 of docs/design/011-pure-transport.md):
in namespaces of its own, the sender links to pure://receiver.test at
127.0.0.2; the receiver is killed, its store, key and memo copied, the
old address made unreachable, the name pointed at 127.0.0.3 and the
copy started there. Every order is applied once, and the moved receiver
makes no X25519.
|
| |
|
|
|
|
|
|
| |
The window, threshold, the start of recovery and CUBIC's four values were
fields of their own, and an episode of congestion events copied each of
the seven aside and back. Now they are one Congestion: an episode keeps
a copy, and undoing it takes the copy back, the window grown since kept.
test-link's results and the M6 simulator's are the same to the byte.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A receiver's run kept seventeen fields of epochs: numbers, flags, secrets
and keys for now, the one before, the one accepted, and an offer waiting.
Now it keeps three epochs (now, before, next), each its number, secret
and the keys of the sender's packets in it, and every move is one of two:
made from a secret, or become the epoch now (the one it was goes before,
the answers are sealed in the new one). Using the one accepted, a chain
begun again from epoch 0, and a run begun from the memo are these moves.
The key of an offer waiting is the key offered: a new offer lets the one
accepted before go at once. A sender's moves (an epoch begun, one taken
for lost, the first) are one too: its packets sealed in epoch n.
test-link's results are the same to the byte.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A sender that took its epoch for lost offers epoch 1 again with a new
key. The receiver accepts it in the turn, after the answers; until then
its answers still carried the ACCEPT of the earlier offer, which named
the same epoch. The sender began its epoch from that one, the two sides
held different secrets, and 2 s later it took the epoch for lost again:
a turn of that each time the order fell so. Now an ACCEPT carries the
first four bytes of the key it answers, and a sender begins its epoch
only from one that answers its own.
test-link cuts the path for 3 s the moment the sender begins an epoch,
three times; a sender's counts say how many epochs it took for lost,
and it must be three. Before, it was four.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
nothing past
A sender offers the public half of a pair made for the next epoch
(OFFER) until the receiver accepts with a half of its own (ACCEPT); their
X25519 and the epoch's secret make the next one's, and the private
halves go. The header's low four bits say a packet's epoch; epoch 0 is
the nodes' static secret, as the keys were. The first epoch is offered in
the run's first packet (it begins with the first answer when the
receiver is not busy), then one an hour or GiB. One who later has both
nodes' keys opens only epoch 0, and no record went there after an epoch
began: test-link tries every packet sent under the static keys, with
epochs a second long, loss, reordering, restarts and replays.
The receiver writes the epoch it accepted in its memo (a place is 256
bytes now), so it goes on without an X25519 when it starts again; one
that forgot is not heard from, and its senders, after 2 s and four
probes unanswered, offer again in epoch 0 with no records until an epoch
begins. Keys of four epochs are tried (accepted, now, before, zero), as
two may share the four bits.
An accept costs an X25519 on each side, 220 µs on this Pi: one of a
sender with records costs a token (256 a second, 8 at once) and the
next packet offers again when there is none; one of a sender with
nothing else to send waits for none, one a second for it. The X25519 is
made after the turn's answers, never between a packet and its answer.
serve's report and M2's generator count keys, X25519s and epochs.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
M2 (ten thousand devices, a record each every 5 s) sent 118k packets for
80k records. A link whose packets were never full still probed for a
1,452-byte path: nothing larger would have been carried. And a sender
said a receiver's challenge back in a packet of its own, one for each
device each time the receiver started. Now a link probes once a packet
of its was full, and a challenge rides in the next packet that goes
(records, or a probe). Until then the receiver's answers there are at
most three times what came, which the answers to one packet are not;
when they are held back, the sender's probe or record sent again says it
back. M2 sends 80,036 again (docs/research/099-link-epochs.md).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
test-link now also plays an attacker: packets changed on the way,
packets of the past sent again from the sender's address and its own,
keys and sources not allowed, and a flood of made-up packets. It found:
- A nonce used twice: a receiver sealed every answer in its one run, and
a sender at rest coming back with the same run got a new slot whose
answers began at number 0 again under the same key. Each slot now
draws the run its answers are sealed in.
- A link stopped for good by one packet of the past: a stream that did
not know its last heard (after a restart or a rest) handed over from
its first record held, and a stale part of an old one never came
whole. It hands over from its first whole record.
- Answers taken away: any packet that opened, however old, moved where
answers went. Only the newest of its run moves it now; a new address
is sent a challenge the sender says back (CHALLENGE, RESPONSE), and
until then gets at most three times the bytes that came from it.
- An answer made by a packet that did not open: a number heard before
was acknowledged before the packet was opened. It is opened first.
- An old answer lowering the sender's credit, or its gap making it send
again: the sender now sends up to the most heard and credit it was
told, and takes a gap only from a word not older than it knows.
- Ten thousand made-up packets made ten thousand keys: keys made for a
run that did not open now cost a token, 256 a second.
Counts now say runs keyed and secrets made.
|
| |
|
|
|
|
|
| |
A sample of 0 (a path within one clock tick, or a test's) made the
pacing rate the window over 0, and its bytes infinity times 0: NaN,
after which the sender never sent again. CUBIC's time-based estimate
divided by it too. The RTT they take is at least a microsecond.
|
| |
|
|
|
| |
Linux's CUBIC and quinn's keep it so; RFC 9438's whole datagram once
past the window before the event is left out.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
acknowledgments
Packets go at the window an RTT, a quarter more (twice in slow start),
up to 2 ms of it at once (10 to 256 datagrams, RFC 9002 7.7); the sender
says when the next may go. The Reno-friendly estimate grows by the
acknowledgments (RFC 9438) or by time (RFC 8312), whichever is ahead, so
a link its app holds back to a few records a second does not fall
behind it.
In the M6 simulator (RTT 100 ms, 5% loss, 50 records a second) the
median time a record takes falls from 134 to 51-71 ms and the p99 from
about 900 to 290-350 ms; a backlog with 1% loss at RTT 20 ms goes at
8.3 Mbit/s. The transport's CPU a record is unchanged.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
acknowledged
A link starts at 1,200 bytes; once an RTT is known it sends a PING padded
to 1,452 (DPLPMTUD, RFC 8899), alone and with the don't-fragment bit,
and uses that size once it is acknowledged. A probe lost is no
congestion; it is tried again a second on, three times, then ten minutes
on. GSO is asked for each burst (a control message), not the socket, so
a packet sent alone is never cut. CUBIC counts in the datagram.
On loopback packets fall by 19-27%; the CPU a record falls 13% at 1 KiB
and 7% at 16 KiB; in the simulator of a 50 Mbit/s path with 1% loss the
goodput rises from 6.1 to 7.0 Mbit/s (5%: 2.3 to 3.0).
|
| |
|
|
|
|
|
|
|
| |
A record that does not fit what is left of a packet goes in part when at
least 64 bytes fit; its parts keep their cuts, so what a packet carried
is whole parts. The receiver keeps the spans it had of a record, and a
record coming in order from its start fills the line's tail with no
allocation. Packets fall by 8% for 1 KiB records and 10% for 16 KiB;
the CPU a record is the same at 61 B and 1 KiB and 5% less at 16 KiB.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A packet called lost stays known a while; acknowledged after all, it
came late: what it carried counts as received, the reordering awaited
grows (later packets acknowledged, and time past the RTT, as RACK), and
an episode of congestion events whose losses all came late is undone.
The window grows only when it was used (RFC 9002 7.8), and congestion
avoidance is CUBIC (RFC 9438), as quinn's.
In a path of 50 Mbit/s whose packets reorder by up to 5 ms, with no
loss, a link carried 1.5 Mbit/s at an RTT of 20 ms and 10 at 100 ms; now
43.6 and 40.1, as without reordering (research/experiments/020-lossy-paths,
virtual time). With 1% random loss: 4.9 to 6.1 at 20 ms, 1.0 to 1.3 at
100 ms. The transport's CPU a record is unchanged.
|
| |
|
|
|
|
|
|
|
|
| |
links.c was a second implementation of the packet format; each device is
now a PureLinkSender on a socket of its own, keyed with a secret made
outside (pure_link_sender_shared: a key kept elsewhere, or a test's
many). M1 (1k, 10k) and M2 (10k, kill -9) pass with it: no silent step,
caught up in 0.2 s, every order applied once; the slowest store after
the restart is now the sender's 1 s wait for a word of the save
(STORED_WAIT), where the old devices tried again after 200 ms.
|
|
|
link/link-protocol.h is the protocol with no socket, thread or clock of
its own: packets and time in, packets out. link/link.h carries it over
UDP as before. A sender starts small (8 flights, 16 records, a 4 KiB
ring) so one thread can hold many. The transport's CPU a record is the
same within 2% (five rounds each way, 61 B to 16 KiB).
test-link runs both sides through a network of its own: loss, packets
held back and doubled, failed saves, an idle sweep, and both sides
starting again, every record applied once as sent.
|
