| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
104 measures a node's memory under a receiver that is down and under a
snapshot held for long (021's run.py: resident memory against what was
posted, 30 s a case). A held snapshot grew the node without end (6.4 to
678 MiB, fixed in d1e8f38); a slow receiver grows it by what was sent,
221 B a send, which is state until delivered, so the budget bounds state
and refuses past it.
013 is that budget's design: what it counts (state bytes, deterministic),
what an event over it does, why an answer takes its send's room, how
links and HTTP wait, what it does not count and those bounds, and the
checks. With --memory 4000000 the same load fills it in about 4 s and
the node stays at 9.6-9.8 MiB (45.9 MiB and growing without).
012 records the hole the budget found in the link (the full flag), and
103 that its clock fix fell short (385754c). CHECKPOINT gains the row.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A link's delivery the app had no room for was handed over again at the
next turn: a receiver over its memory budget would refuse the same
records as fast as the owner could apply them. And its sender, seeing
records received but not stored for a second, sent the whole window
again every second.
Now a host says a delivery found it full (PureLinkDone.full). The
receiver keeps the records and hands them over again after a pause, 100
ms doubling to 500 ms, and meanwhile its STORED frames carry flag 2: the
app is full and what came is kept. A sender hearing it starts its
one-second wait for the save again, and sends none of it again. The
spec (012) says both sides; the pause is a local choice that must stay
under the sender's wait.
pure_link_done takes the PureLinkDone itself. test-link's new case keeps
the app full for 5 s: 12 hand-overs meanwhile, taken within 0.5 s of
having room, and no record sent again (21,640 without the flag).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
102: the spec (012) and a receiver written from it alone; the four holes
writing it found (REFUSED, the record's size, a peer's run places, the
memo's places beside) and the credit M6 showed fixed at 1,024 records;
M6 with ngtcp2 as a third, two rounds. Under reordering Pure carries the
most (45 Mbit/s; ngtcp2 29-47, quinn 1-2), under random loss the three
are the same, on a clean bottleneck Pure carries 3-5 % less (its framing,
some 112 B a packet to QUIC's 41), and where loss and reordering meet
ngtcp2 carries 1.3-3.6 times as much: undoing a loss it took for
reordering, it restores the window a real loss had cut too (its log: 22
of 34 undos past the window before the last cut; none without
reordering). Shares beside TCP stay within 25 %.
011 and the status table say so; 012 also says that a receiver answers
within 25 ms (a sender's PTO counts on it), that a shape and a record's
bytes are the app's, and why a packet of the past does no harm.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A receiver gave every stream a credit of 1,024 records past its last
stored, however small the records: a link carried at most 1,024 records
a round trip. On a path of 100 ms that is 10,000 records a second of
61 B, a ninth of what 50 Mbit/s carries (the path simulator: 9,953 a
second, 90,814 at 10 ms); and records of 1 KiB, a queue's worth of them
in flight, went a record or two a packet as each STORED let a few more
through (M6: 107,968 packets where 20 ms took 84,106).
A receiver bounds what it holds of a stream (16 MiB) apart, so its
credit is now what fits there at the size its last delivery's records
had: 1 to 65,536 (what a sender takes), 1,024 before any delivery. It
takes records that far past its last stored, as its memory lets it.
Simulator at 100 ms: 56,296 records a second of 61 B. M6 at 100 ms:
45.4 Mbit/s of 1 KiB records in 83,566 packets (44.9 in 107,968). The
window, not the credit, now fills the bottleneck's queue: losses there
where the credit hid them (61 B at 10 ms: 87,130 a second, 1,371 lost).
012 says so.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A receiver's memo kept one run's epoch a peer. A node whose outputs
send to this one sends in a run each, so after the receiver started
again (or let the peer rest) all runs but the last accepted were told
LOST and began their epochs again: records under the static keys for a
round trip, and an X25519 each. Now a run whose peer's place holds
another run still sending keeps its epoch beside, in one of two of
1,024 places its peer and run find (160 KiB at the file's head: the
memo is PLM3, and a PLM2 one is begun again). A run's place beside goes
once its peer's place has it, so an epoch's secrets stay only until the
next one's are.
test-link's three outputs of one node, each in epoch 1, go on after the
receiver starts again: no LOST, no X25519 (before: two LOSTs, an
accept). 012 says so.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
012 says what goes on the wire, enough for another implementation: the
header, the key id, the version and why it is not negotiated on the
wire, the static secret, HKDF's salt and infos, the nonce, epochs'
chain, the thirteen frames byte by byte, what each side must do, the
values that are the spec's and those each side picks, the two words a
receiver says (LOST, REFUSED), what an attacker gets, and what is known
not to hold. A receiver written from it alone, in Python, links with a
pure-serve node through epoch 1 (check.py).
011's last row of its stop table is met: H1-H4 held, every check passes,
CPU within 10 % of QUIC at 61 B and 25 % under it from 1 KiB, the share
with TCP within 25 %, 3,097 lines of C. Links without connections are
now the default between Pure nodes; HTTP stays the boundary to others,
and relayed sealed sends stay as they are. 006, 009 and the README say
so.
|
| |
|
|
|
|
|
|
|
| |
A path MTU black hole is seen and the link goes back to 1,200 bytes; a
receiver that lost its memo says LOST and M2 without a memo catches up
in 0.4-0.5 s, not 7.7; M5 stands beside ngtcp2 too (Pure 25 % lighter
than both QUIC from 1 KiB, 1.04 of ngtcp2 and 0.90 of quinn at 61 B);
and the stream's name in every packet, 096's guess for 61 B's gap, cost
nothing measurable, so it stays. 098 notes the one line LOST changes.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
H3 of design 011 stands: with ten thousand devices sending 2,000 records
a second, the node's store, key and memo carried to another address,
nothing stopped for a 100 ms step, the moved node made no X25519, and
every record was applied once. Each device's first record after the move
waits one probe (30-45 ms here), as its first packet finds no way there.
The search found the listener answering from an address other than the
one a packet came to, and a link that looked for its receiver after its
probe went, losing every answer. H1-H4 are all measured now; 61 B's CPU
a delivery is at the edge of the criterion, to be looked at before step
5 (adoption and a specification).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Forward secrecy, 098's first task left: keys that leak later open only
epoch 0. M2 after the epoch began and after a restart is as before it
(p50 0.4 ms, p99 0.8 ms, the same CPU, no X25519 on restart). It costs a
p50 of 0.7-1.0 ms more while ten thousand devices first meet and for
about 40 s of accepts after, and a receiver that lost its memo now
stalls 0.8 s and catches up in 7.7 s, not 0.2-0.7: it lost the epochs,
and each sender opens one again. The search also found a packet more a
device from each of M6's path probe and the security checks' challenge,
and a load generator whose own pass over its devices was the floor of
the latency it measured. 098's note that epochs narrow KCI to the first
epoch was wrong; corrected.
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
(097)
M6 of design 011: with a 50 Mbit/s bottleneck, a link carries 21-41
times quinn's goodput when packets reorder, the same under random loss,
and 1.3-3.5 times with both; its p99 at 50 records a second is the same
or shorter; beside TCP CUBIC it takes 0.48-0.55 of the path, inside the
guardrail. Only on a clean path is it 3-5% below quinn.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
M5 of design 011. At first a link spent eight times quinn's CPU on a small
record: scans of the backlog, a system call and allocations a record, 19
bytes of framing. After the seven changes before this one, the transport
alone (transport.c; a backlog drained, every record checked) spends per
record 0.30 us for 61 bytes (quinn 0.27: 1.06 to 1.12 times over three
campaigns, at the design's 10 % line), 2.91 us for 1,013 bytes (quinn
3.88) and 45.2 us for 16,013 bytes (quinn 60.9): 25 % less for the larger
ones, and 30 % more records a second. What is left for small records is
per packet: the stream's names and the RECORDS head. quinn's numbers carry
no confirmation of storing and no exactly once.
Between two pure-serve nodes the links spend more per record, woken at
the receiving app's pace. Filling the sender showed the outbox is written
whole at every save, as the numbers heard were (094): the next thing to
change.
run.py gains --transport and --drain; the quinn service reads a drain a
chunk at a time and its client writes 64 KiB at a time.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
M2 and M4 of design 011. Ten thousand devices send an order every five
seconds (2,000 a second); the receiver is killed (-9) at about 21 s and
started again at once. With its secrets remembered the node shows no
100 ms step without an order stored, catches up in 0.2 s, spends no more
CPU than before and makes no X25519; 34 orders were sent again, and
every order made was applied once (three rounds; on the SD card the
same, with 0.2 s silent). Without the memo it makes 10,000 X25519, 1.6 s
of CPU. quinn under the same load stops for 1.0 s and catches up after
10.9 s: 10,115 new connections, about 7 s more of the service's CPU and
8 s of the clients'. A first contact takes Pure one round trip and QUIC
two, at 10, 50 and 100 ms of RTT.
run.py --restart and --first-contact make them; the quinn harness gains
a fixed port with a reset key from its key, clients that connect again,
and a drain for M5.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
M1 of design 011. A Pure node allows 1,000 to 1,000,000 senders; each
sends one order and, once told it is stored, rests. At rest a link costs
the node 132 bytes (the sender allowed, 73; its number heard, about 60)
and no transport state; with a million the node runs in 139 MB. A quinn
0.11 service holding the same exchange's clients connected at rest costs
38,106 bytes a connection: 289 times as much, past the design's 1/100.
A link at rest takes its next record in 0.56 ms to stored, one X25519 on
the node included; a new QUIC connection takes 1.06 ms and 2.8 times the
receiver's CPU, without storing.
research/experiments/019-idle-links: links.c, many senders written from
serve/link.h's packet format alone (a second implementation), and the
quinn service and clients (quic/, built offline); run.py makes both
measurements.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pure-serve --send OUTPUT=pure://host:port/INPUT sends an output's saved
sends to another node's --link-port with no handshake: the first packet
carries records. Keys come from the two nodes' keys (X25519, HKDF-SHA256,
AES-128-GCM), with a random run in each packet's header so no nonce
repeats across restarts; nothing is stored for them. What lasts is the
apps' own: the sender's outbox and the receiver's heard numbers
(pure_app_heard, new). The sender detects losses as RFC 9002 says, with
NewReno, and trims a send once the receiver says it stored it; the
receiver hands the owner whole records in order, which it applies from
the one after the source's last heard number, stopping at a gap, and
answers how far it stored with every acknowledgment. Stored bytes are
the bytes on the wire: an output and an input of one shape need no JSON.
server_links_udp kills the receiver, the sender, then both (-9) while
four clients buy: every saved order is applied once, with no handshake
(five runs, 10k-11k orders each). server_links_udp_lossy does it again in
a namespace of its own with the link's packets delayed, lost (3 %) and
reordered (10 %) by netem (three runs, 11k-12k orders).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The first draft replaced a few of QUIC's layers with what Pure already
has, every technique known and the gains constant factors, and it kept
the connection. This one removes it. Nodes have keys, links are the app's
wiring, paths are a cache of where the last authenticated packet came
from, and a link's transport state (key epoch, stored-through number,
heard numbers, packet-number reservation) is a row committed atomically
with the app's state; memory holds only active links and per-path
congestion state. So a restart of either node is invisible to the other
(no handshake, nothing lost or doubled), an idle link costs a row on
disk rather than process memory, and moving the store moves its links.
Forward secrecy comes from key epochs rather than per-connection
handshakes; replay protection survives restarts because heard numbers
are stored with the events they applied (OSCORE, the closest prior art,
needs an Echo round trip there).
Hypotheses H1-H4 and a measurement plan against quinn and ngtcp2:
memory per idle link at 10k to 1M links, kill -9 under load with 10k
active links, moving the store, first contact, and CPU, loss and fairness
as guardrails; stop if an idle link is not at least a hundred times
cheaper than a QUIC connection, if restarts need handshakes, or if
anything is lost or doubled. Beyond it: speculative execution of
verified, deterministic modules, for interactions that do not wait.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Outside feedback 002 asks for a protocol of Pure's own that beats QUIC.
Pure's delivery is already stored, numbered, applied once, and sealed per
link; QUIC and TCP with TLS own the same things again (retransmission
buffers, replay protection, encryption, flow control, acknowledgment).
The design takes the end-to-end argument literally: the sending app's
store is the retransmission buffer, one packet carries both the packet
ACK and the stored-through number, a direct link is sealed once with the
nodes' session keys, first contact carries records in its first packet
(the receiver's key is wired in advance and applying once makes replays
harmless), credits come from what the next save can take, the session is
known by its key rather than its addresses, and stored bytes are the
bytes on the wire. Loss detection, congestion control and cryptography
stay standard (RFC 9002, NewReno then CUBIC, HPKE), plain UDP sockets
rather than SOCK_RAW, HTTP stays the boundary to others.
Written before any code: the decision (this transport, QUIC with Pure's
format, or neither), the open questions, the smallest checks, a
measurement plan against quinn, ngtcp2 and the current HTTP delivery
(seven workloads, RAM and SD stores, loss and delay from unprivileged
namespaces with netem, checked to work on this Pi), and stop criteria:
no transport if it is under 30 % of a delivery's CPU; stop below 1.5x
the best QUIC per CPU; adopt at 2x with a round trip fewer, a quarter of
the memory per idle link, fair to TCP, every correctness and security
check passing, and under 5,000 lines.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A fold whose kind has PURE_FOLD_SLICED (128), and a walk whose argument
has PURE_EACH_SLICED (bit 0), first take two numbers, a low and a high
key, and run for the items of their list (keyed by a number) whose keys
are from low up to before high, found by key: the range is the program's
meaning, not an optimization, so their blocks may fault (items out of the
slice never run). A fold a leaf at a time now leaves out the lanes of a
leaf outside the items it runs for, and a leaf that would fault runs item
by item over those items alone. Slices are neither shaped nor given
automatic ranges.
ranges now checks a count over a slice whose test divides (a division by
zero failing it only when inside the slice) against a model, the same on
the item-by-item path, and a walk over a slice.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A filter, a fold or a walk's where (a test at the start of its block and a
jump to its end when it fails) that ANDs comparisons of a keyed list's key
with values reading no item gets a range at load (PureRange: a lower bound
GT or GE, an upper LT or LE, EQ both). When it runs, the bounds are found
once and looked up by key, and only the leaves between the two positions
are read: O(log n + k) where it read every item. A filter leaves the items
out of range out anyway, so it always may; a fold or a walk only when the
rest of its test and its value cannot fault (no division, sum or product
that may overflow, key or item that may be missing), since the items it
skips would have run them; a walk's bounds must also read no state, local
or list its block writes, as the where reads them anew for every item.
A bound that faults leaves the test to run for every item, as written.
A fold of kind 5 (distinct) counts the different values its included
items give (-0 as 0), with a set per fold; it runs a leaf at a time and
is never shaped. max, min and first alone fault over no item.
ranges checks, against the same module reading every item, over 60 lists
of up to 400 keyed items and 3,600 views: a count between two bounds, a sum
between a lower and an upper bound with another condition, first equal to
a key, min with the bound on the right, a count whose test may fault (no
range), a filtered table in bounds, 359 walks changing and removing the
items in bounds, and a walk whose body moves its bound (no range).
lane_folds now makes distinct folds too.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
LOCAL_LOAD and LOCAL_STORE name one of an event's 64 slots, what it holds
(a number, or an index of a list) and that list: slots start at 0 when an
event begins, are shared by its handlers, the walks in them and the folds
in those, and are never saved nor in the states' namespace. A handler uses
a slot for one thing, so its type is verified as a state's is; views have
no locals (IN_LOCALS: a handler's block, a walk in it, a fold in either;
stores only in a handler and its walks). A fold reading a local still runs
a leaf at a time (one value for every item) and a shaped fold takes one
only on its key's side. The signature table's where widens to 16 bits.
locals checks a handler that sums items into a local in a walk, counts in
a fold the items below it, keeps an index in a local and reads it back,
over 80 events against a model, and five refusals. server_rows' shop now
takes stock through an index local and counts in one.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
GET /views/name?argument=value gives a view that takes arguments its
arguments: the query as it came is kept on the request's work slot (it
outlives the connection, which may go before the view is shown), and when
the view is shown, on the connection thread from a snapshot or by the
owner, its pairs (values %XX decoded) are laid out as a payload of the
arguments' shape and given before the view is evaluated. from and count
still page a table. An argument missing, unknown, given twice, not of its
type or badly escaped answers 400 invalid. A registered query takes its
view's arguments the same way (GET /queries/name?k=4). An arguments input
is not an input to post to.
server_arguments opens 200 accounts, then asks 83 times against a model:
balances by key (404 no_value for a key not there), counts above an
argument, texts with &, =, % and non-ASCII shown back, a table filtered by
two arguments whose column folds with one, paged, counts by an owner's
name, a registered query with an argument, and six refusals. The format's
document (008) describes arguments.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A rows field's JSON value is an array of objects naming the record's
fields (numbers, strings; a field left out takes its default). The
connection thread encodes them as the list's stored form, which may be
longer than the JSON, grows the request's input by that much
(pure_runtime_grow: charged to the payload budget as admission charges an
input, 503 payload_budget when it does not fit) and places them after the
body, so the owner still applies a message it need not parse. A message
whose only variable field is rows takes a body that is not JSON as those
rows in their stored form: a bulk insert without JSON. An API's answer
(--reply) gives rows the same way. A delivered batch holds no rows.
server_rows loads 30,000 stock rows in one JSON request (some 250,000
rows a second, against 6,500 with a request a row) and 20,000 more as
bytes, refuses a load with a key already there whole (409 exists), then
checks 400 random orders against a model (294 applied, 106 refused with
none of their rows applied), texts with escapes and Korean, a walk that
removes, and a restart that keeps every state and no rows.
The format's document (008) describes walks, rows fields and lists of
rows, and OUTER_ROW.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pure-serve --surface LIBRARY [-- ARGUMENTS] maps a library and runs its
pure_surface_run in a thread of its own (serve/surface.h). The surface
reads the app as of the last save where it is, a published snapshot in a
reader app, as a connection thread reads views; an eventfd tells it of
each new snapshot; and its events (a key pressed, text typed) are
requests without a host in the owner's queue, applied and saved with the
round, their results handed back to the surface's waiter. So one node
serves programs over HTTP and people on its display with one owner of
its states, and nothing between the states and the screen. pure-serve
exports its pure_* functions for the surface to call, one copy of the
app model in the process; the table carries sizeof(PureApp) and
sizeof(PureModule) so a surface built for another layout refuses.
The check builds a probe surface that types, presses and reads: its
events show in HTTP views, an HTTP input shows in it, wrong inputs are
refused, it stops with the server, and a module without states cannot
have one. Without a surface the request path measured the same (small
GETs 140.9k vs 139.9k, app totals 126.3k vs 126.3k over eight rounds);
the exported symbols cost 11 KB of the file and 24 KiB idle.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Linking OpenSSL cost every pure-serve 2.3 MiB of resident memory, 0.66
MiB of it the process's own (relocated data), though most servers never
reach an https receiver or seal a send. The code that uses OpenSSL (TLS
to https receivers, sealing) is now built as pure-crypto.so beside
pure-serve and exports one table; pure-serve maps it the first time a
wire to an https URL opens or a key is read or a seal wired
(serve/crypto.c), and a missing library is reported at startup. What
needs no OpenSSL, a key's hex and the suite this CPU seals with, stays in
pure-serve.
An idle server: RSS 4,496 -> 2,320 kB, PSS 1,639 -> 881 kB, private
dirty 1,296 -> 864 kB. A static OpenSSL was measured too and is worse
(RSS 3.8 MiB, but 2.3 MiB private: all of its tables relocated). The
checks, https webhooks and sealed sends included, pass unchanged.
|
| |
|
|
|
|
|
|
|
|
| |
The HTTP host's end-to-end checks (documents against the earlier server,
lending, kill -9 durability, JSON, timers, links, webhooks and answers) sat
in aup's tests, so a change to the host was verified from the toolchain.
They now live in tests/check.py; make check builds their apps with the
sibling aup and runs them after the unit tests, which stay beside their
parts. Design 009 states the rule for all three repositories.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
The benchmark runs an earlier pure-serve build in turn with the current one
(--baseline). Against the build before the move, linking through libpure.a
costs nothing: a one-command build with the library's object order has the
same size and the same CPU per request. What moved was code placement, so
the runtime links last again, as it did; the two orders now differ by less
than rounds of one build do (total -0.4%, page +1.2%, mixed -0.1%). The
binary is 5,008 B smaller: unused flow.c functions stay in the library, and
the linker's A53 erratum padding is gone.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The app model, the tree store and the HTTP host lived in
prototypes/connections among the earlier document runtime, the store was
known only by its format's name, and Vega compiled six Pure sources by path.
Now, as the README's principle has it:
core/ runtime, flow and the app model (module*.c), with what the core
asks of storage (state.h, store.h)
store/ the tree store (PURETRE3, PUREREC1)
serve/ pure-serve: HTTP host, outbound delivery, JSON in place
cli/ pure-run, pure-edit-module
A top-level make builds libpure.a (app model and store, for hosts to link)
and the hosts; make check runs the store's and the app model's tests. The
earlier runtime still builds from prototypes/connections against core/ and
store/ (check, check-tree and check-core pass). Design 009 records the rule:
parts by role behind libraries; git repositories split by coupling, so the
store stays here while its contract moves with the core, and Vega stays apart.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The final answer to each send of a wired output (a 2xx, or a 4xx but
408 and 429) becomes a message event of the given input: fields from
the answer's JSON object by name, other members skipped; a field the
answer lacks from the send's field of that name and type, so the
request's own id is there; a u32 http_status field is the status. It is
applied with the send's number from the wire's source, before the send
is trimmed and in the same save, so an answer that comes again after a
restart (the API repeating itself for the Idempotency-Key) applies once.
Sends that wait for answers go one per request; answers are saved with
the next round, or at once when none waits.
A server that writes an answer in pieces holds the second until the
first is acknowledged, and a delayed acknowledgement stalled every
answer 40 ms: the client now acknowledges at once (TCP_QUICKACK) after
each read. Against a payments API that answers 200 or 402, with the
sender killed (-9) midway, paid and declined orders matched the API's
answers one to one.
|
| |
|
|
|
|
|
|
|
|
| |
A wire to an https:// URL speaks TLS 1.2 or later through OpenSSL: the
receiver's certificate must chain to the system's CAs (or to the file
--send-ca names) and name its host (SNI and SSL_set1_host, or the IP
address). An untrusted receiver gets nothing: the send stays in the
outbox, the wire logs it and tries again every 10 s. pure-serve now
links libssl and libcrypto.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pure-serve --send OUTPUT=http://host:port/path wires an output: after
every save the owner queues the new sends as JSON (fields and groups)
and a thread per output posts them in order with X-Pure-Source,
X-Pure-Sequence and Idempotency-Key. A 2xx, or a Pure receiver's
refusal, delivers; the owner then trims. 408, 429, 5xx and no answer
are sent again after a pause growing to 10 s, other answers every 10 s:
nothing is dropped. A Pure receiver applies a delivered message with
pure_app_deliver (now taking placed fields) and advertises
X-Pure-Batch: 256; the sender then posts up to that many numbers in a
row as one JSON array (X-Pure-Count), applied in order in one burst
with one save.
Delivery kept up with orders once batched (391 deliveries/s one at a
time; a backlog of 20,000 went at 60k/s). With four clients ordering,
killing the receiver and then the sender (-9) left the receiver's count
and total equal to what the sender saved, five times and under ASan.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
An output (input kind 5) is a message the app sends; SEND (72) takes
an output's field values in order in a handler, lays them out as a
message's payload and keeps it, numbered per output from 1, with the
event's changes: a failed event sent nothing, and a failed save takes
back every send since the last save with its numbers. A host sees only
saved sends (pure_app_outbox) and trims what its receiver has saved
(pure_app_trim, gone with the next save). pure_app_deliver applies a
message from a named source once: a number heard before succeeds
without applying, a refused message counts as heard, a failed save
forgets what it heard.
Sends, numbers and heard numbers are the app's links state, stored
whole in one runtime document (PLK1) with the states, in the same
commit. Outputs follow their names across module changes; a module
that drops or retypes an output with sends left is refused, and a
dropped output's number is kept for when it returns. Killing the
process at every commit stage leaves an event and its send, or a
delivery and its effect, together or neither. pure-serve answers a
post to an output or a timer with 404 no_input.
|
| |
|
|
|
|
|
|
| |
One connection thread for each CPU the process may run on (its
affinity, or the online CPUs), less the one the owner of an app with
states takes: three CPUs give two connection threads and the owner.
The default stays one thread.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
After a round that saved, the owner waited for 100 us without a new
request (1 ms at most) before the next round. Requests that queued up
during the save made that quiet start at once, so the clients just
answered missed the next round; the clients then settled into two
halves taking turns, each save carrying half of them. It showed with
two connection threads: 34 events a round and 15.8-16.0k posts/s with
64 connections, against about 61 and 22-24k/s with one or three. The
quiet now counts only once a request has come after the answers were
handed back: two threads carry 61.6 events a round, 22.9-23.9k/s.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pure-serve no longer copies a JSON body's strings into the message's
payload. The ingress is the prefix, a table of each field's start and
size, and the body as it came; the connection thread scans the body
once, puts numbers into the prefix and records where each string is.
Only a string with escapes is unescaped, over itself (it never grows).
A group of fields comes as an object inside the body and, for a fixed
field, as its path in the query (?to.zip=7). A group where a field
belongs, a value where a group does, an array or true/false/null is
json_type; an unknown member, inside a group too, json_field.
The string scan and UTF-8 checks now pass over eight bytes at a time
while none needs a look (pure_utf8_step, so every text check gains).
On the Pi, posting {"name":"k","text":"x..."} to a stateless app went
from 652-655 to 3,737-3,953 requests/s with 256 KiB of text (server
CPU 1.52 to 0.25 ms a request, peak RSS 5.2 to 3.2-3.4 MiB) and from
29.1k to 76.3-76.7k/s with 4 KiB; the load generator gains a json
method and benchmarks/http/echo.aup is the app.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A message field's name may now be a path of identifiers joined by dots
(user.name, a field inside the group user); a module that names both a
field and a group (user and user.name) is refused, as are empty,
leading, trailing or digit-first segments. A module may have 64
message fields instead of 32 (PureModule 11.9 to 13.2 KiB).
pure_app_message_fields takes each variable field's start and size in
the payload, so a host that parsed a request in place passes the
fields' bytes where they are instead of laying them out after the
prefix. The runtime checks them as before: a field past the payload
fails with limit, a text that is not UTF-8 with encoding.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The owner publishes a snapshot of the saved states after every save,
before it hands back the answers; connection threads take the newest
one each round and show views from it with a reader app of their own.
A read made after a write's answer sees the write, nothing unsaved or
undone is ever shown, and reads no longer wait while the owner applies
events or syncs. Apps with states always get an owner thread, even
with one connection thread. A round's inputs go to the owner first;
its views are shown and then answered together (answering each one as
it was shown made one thread's 20-row page 8% slower).
On the Pi 5, with 8 connections posting durable scores and 56 reading
the total, reads went from 10-16k/s at p99 13-29 ms to 107-149k/s at
p99 0.6-0.7 ms, and posts stayed in range; axum and drogon, which sync
under their lock, read 1.1-1.3k/s at p99 111-156 ms. A snapshot of a
changed 1,024-item list costs 1.2 us (65,536 items: 87 us) against a
3-4 ms save.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Right after a save, pure_app_snapshot makes a read-only snapshot of the
saved states: numbers are copied, text states are frozen drafts that
share the store mapping (pure_draft_freeze, no copy), and lists are
copies of their item pointers that share the items. An unchanged list
shares the last snapshot's copy. pure_app_read binds a reader app on
another thread to a snapshot so the existing view functions read it.
A shared item never changes again: a list stamps each item with its
share count, copies a shared item it changes, and retires shared items
it lets go instead of freeing them. Snapshots form a chain that goes
oldest first, and items retired while a snapshot was the newest go
with it. References are atomic and may be released on any thread.
test-module checks that each snapshot equals the app, that kept
snapshots stay as made through changes, removals, a module switch, the
app stopping and the store closing, that a failed save's snapshot
equals the one before, and that two threads read the newest snapshots
while the app goes on (also under ASan, UBSan and LeakSanitizer).
|
| |
|
|
|
|
|
|
|
|
|
|
| |
A view that shows a text state as it is (pure_module_text_state) now goes
out as the state's stored extents, as a document read does: the answer
holds the store mapping, charged to the budget, until it is sent and its
ticket released; a state with edits not yet stored is copied as before.
Documents and views share one loan path, and the extent array grows as
needed. A 1 MiB note read by 8 connections: 776-817 answers a second
copied, 1,875-1,901 lent, p50 9.2 to 3.5 ms, peak RSS 10.3 to 5.7 MiB.
The load generator learns GET /views/page.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
A real app (a board, a stopwatch) already used a quarter of the 512
instructions, since a fn is inlined where it is called, and every text
it shows is one of 64 constants. The code and the constants now live in
one block sized to the module, which load takes and pure_module_release
gives back; the limits rise to 65,535 instructions and 4,096 constants
of up to 1 MiB, while PureModule shrinks from 28,664 to 11,936 bytes.
The verifier's branch marks are sized to the block instead of 64 KiB on
the stack. pure_module_read reads a module file whole, and every host
uses it instead of a 128 KiB buffer.
|
| |
|
|
|
|
|
|
|
|
| |
pure-serve sets the time an event sees to the wall clock when the event
is applied, and the thread that runs an app with states (the connection
thread alone, or the owner) ticks its timers on time: epoll waits no
longer than the next tick, the owner's condition wait is timed to it, and
the ticks due at once share one save, so a restarted server goes on from
the stored ticks. The statistics count ticks.
|
| |
|
|
|
|
|
|
|
|
| |
An input may be a timer (kind 4, its period of 1 ms to a day in place of
a key name): pure_app_tick delivers a tick, an event saved like a key's.
NOW gives handlers the time the host set in PureApp.now (milliseconds
since 1970, UTC), so a script replays any time and views stay functions
of state; FLOOR and MOD take times apart, MOD taking the divisor's sign as
floor division leaves it, a zero too. pure-run gains time MS and tick NAME.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A message field is now a u32, a number (binary64), bytes or text, and
variable fields may be several and anywhere: the payload's prefix holds
the fixed fields and each variable field's length but the last one's,
which takes the rest; a module whose one bytes field came last is laid
out as before. On arrival a length past the payload, a text that is not
UTF-8 or a number that is not finite fails the message.
pure-serve fills an input's message from a JSON object body
(Content-Type: application/json) on the connection thread before the
request is ready, so the owner never parses and replies are still
borrowed: the ingress is [u32 payload size][payload][the JSON as it
came]. A missing, unknown, doubled or mistyped field, or text that is not
JSON, answers 400 and the request never runs. Numbers may come in the
query too.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
The scoreboard, 22 lines of aup served by its views and inputs, against
the same service written by hand in axum and drogon on the Pi: durable
posts about 76 times as many, reads level with drogon and ahead of axum
with the least CPU and memory. The note keeps what the profile showed on
the way (the kernel's share of a single thread, integers through printf,
an owner that saved too early) and what comes next. run.py gains
pure-serve-2 and waits for a server that prints its port before it
listens.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
States have one owner, so a module with them ran on a single thread,
which spent most of its time in the kernel sending and receiving. With
--threads N such a module now gets N connection threads and one owner
thread: connection threads accept, parse and answer; the owner takes the
ready requests of every thread as one round, applies them in bursts with
one save each and hands each thread back its applied requests (eventfd),
which it completes and answers. A request's route and payload stay held
by its thread's runtime core until completion, so an accepted event runs
even when its connection closes first.
After a burst that saved, the owner gathers requests until none has come
for 100 µs (1 ms at most), so the requests its answers bring back share
the next save as they would on one thread: 30 events per save became 50
and the median latency of a durable post fell from 5.4 to 3.5 ms. A page
of 20 rows went from 87,600 to 141,900 answers a second on the Pi.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pure-serve answers GET /views/NAME with a view's value (a number or text
as text/plain, a table as JSON rows, a page of them with ?from=&count=,
each cell computed only for the rows asked) and POST /inputs/NAME with an
event: a text input's text is the body, a key input is pressed, a message
takes its u32 fields from the query and its bytes field from the body and
answers its handler's reply. A failed event answers with its fault (404
not_found, 409 exists, 422 the others). No routes are written: the app's
views and inputs are its interface. Documents stay served when the module
has show and edit.
The route of a request is kept per work slot, so an accepted event runs
even if its connection closes before the burst; views and app events join
any burst, whose one save comes before its answers.
|
