| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
M5 of design 011. At first a link spent eight times quinn's CPU on a small
record: scans of the backlog, a system call and allocations a record, 19
bytes of framing. After the seven changes before this one, the transport
alone (transport.c; a backlog drained, every record checked) spends per
record 0.30 us for 61 bytes (quinn 0.27: 1.06 to 1.12 times over three
campaigns, at the design's 10 % line), 2.91 us for 1,013 bytes (quinn
3.88) and 45.2 us for 16,013 bytes (quinn 60.9): 25 % less for the larger
ones, and 30 % more records a second. What is left for small records is
per packet: the stream's names and the RECORDS head. quinn's numbers carry
no confirmation of storing and no exactly once.
Between two pure-serve nodes the links spend more per record, woken at
the receiving app's pace. Filling the sender showed the outbox is written
whole at every save, as the numbers heard were (094): the next thing to
change.
run.py gains --transport and --drain; the quinn service reads a drain a
chunk at a time and its client writes 64 KiB at a time.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A record over a piece (1,000 bytes) went in pieces even when it fit the
packet whole: a record of 1,013 bytes went as two RECORD frames, and the
receiver gathered it the slow way (an allocation, pieces, then onto the
line). A record none of whose pieces went yet now goes whole in a RECORDS
frame whenever it fits what is left of the packet; its pieces are marked
sent together. A record too large for a packet, or one resent after
some of its pieces were lost, still goes in pieces, and a receiver takes
a record whole or in pieces alike.
The transport alone takes 2.85 us a record of 1,013 bytes, not 3.18
(quinn: 3.90). transport.c now checks every record delivered: in order,
once each, its bytes as sent. None was wrong in a network namespace with
1 ms of delay, 3 % loss and 10 % reordering, with records of 61, 1,013
and 16,013 bytes.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
A queued record had an allocation of its own for its bytes and its
pieces' states, freed when the receiver stored it. Records come and go in
order, so their bytes now go one after another into a ring, never cut by
its end, and go back as the ring's head moves past the ones stored; a
ring too small grows to twice its size, the records moved into it in
order. A record of one piece keeps its piece's state in itself.
The transport alone drains 200,000 records of 61 bytes at 5.3 to 6.5
million a second; per record the link takes 0.137 us, the listener 0.14:
0.28 in all, against quinn's 0.26 a message over one stream.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
A receiver copied each record into an allocation of its own, copied them
all again into a delivery, and freed each once stored: about 110 ns a
record of 61 bytes. Once the last number heard is known, whole records
that come in order now go onto the stream's line as a delivery carries
them (u32 size, bytes). The line is handed over as it is, and let go by
moving its head; its front is given back when there is no room at its
end. Records that come early or in pieces, or before the last heard is
known, still wait in held, and move onto the line once it reaches them.
The transport alone: the listener took 0.20 us a record of 61 bytes, now
0.15; with the link, 0.31 us a record in all, against quinn's 0.26.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
changes
A listener's socket now asks for GRO: the kernel hands over the datagrams
of one sender that came one after another as one buffer, each the
segment's size but the last, and on the loopback a burst sent with GSO
comes whole instead of cut into datagrams. The listener takes up to eight
such buffers at once. It gives its AES-GCM context a key only when the
packet's run has another than the one it holds, for opening and for
sealing answers.
The transport alone drained 200,000 records of 61 bytes at 4.2 million a
second (0.454 us a record, both sides); now at 4.9 million (0.394 us).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
A RECORD frame carries a number, a size, an offset and a length before
each record's bytes, 19 bytes: a packet held 13.7 records of 61 bytes. A
RECORDS frame (0x06) carries the first record's number and a count, then
whole records whose numbers follow on, each its size as a varint and its
bytes: 17.2 a packet. A piece of a record larger than a piece still goes
in a RECORD frame; a receiver takes both.
The transport alone (research/experiments/019-idle-links/transport.c)
drains 200,000 records of 61 bytes in 11,600 packets instead of 14,600;
per record the link takes 0.217 us and the listener 0.23, 0.447 us in all,
against quinn's 0.43 a message over one stream.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Every queued record wrote the link's eventfd, a system call each, and woke
the thread to send a packet of a few records. A record queued now wakes
the thread only when it sleeps and nobody woke it yet; one queued while it
sends marks the link, and the thread looks again before it sleeps. With
records queued one at a time as fast as a thread can, 200,000 small ones
went at 400k a second, and now at 730k; queuing one costs 0.43 us, not
1.0.
research/experiments/019-idle-links/transport.c measures the transport
alone: a link and a listener in one process, no app, no store. A backlog
of 200,000 records of 61 bytes drains at 3 to 4 million a second, 0.24 us
a record on each side.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A backlog made the link slow in its square: the sender looked at every
queued record's pieces twice a turn to know whether any waited to be
stored, and the receiver moved all its held records down each time it let
the first one go.
- The sender counts the pieces received and not stored, and keeps where
the unsent ones begin; every change of a piece's state goes through one
place that keeps both.
- The receiver keeps its held records from a head index. One in order goes
last, and one out of order finds its place by halving. The record's bytes
and its pieces' marks are one allocation.
- Packets go in bursts with UDP GSO, 54 at a time: 0.45 us a packet on this
Pi, against 2.4 us for a send each. A full packet is padded to a whole
datagram (PADDING frames, as QUIC does). Datagrams are taken 64 at a time
(recvmmsg).
Twenty thousand saved sends of 61 bytes drained at 370k a second; now at
990k. Per record, the sender's link thread took 1.1 us and now takes 0.5;
the receiver's took 2.4 and now takes 0.5.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
M2 and M4 of design 011. Ten thousand devices send an order every five
seconds (2,000 a second); the receiver is killed (-9) at about 21 s and
started again at once. With its secrets remembered the node shows no
100 ms step without an order stored, catches up in 0.2 s, spends no more
CPU than before and makes no X25519; 34 orders were sent again, and
every order made was applied once (three rounds; on the SD card the
same, with 0.2 s silent). Without the memo it makes 10,000 X25519, 1.6 s
of CPU. quinn under the same load stops for 1.0 s and catches up after
10.9 s: 10,115 new connections, about 7 s more of the service's CPU and
8 s of the clients'. A first contact takes Pure one round trip and QUIC
two, at 10, 50 and 100 ms of RTT.
run.py --restart and --first-contact make them; the quinn harness gains
a fixed port with a reset key from its key, clients that connect again,
and a drain for M5.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
A listener lets a quiet sender's secret go with its active state, so a
node started again made one X25519 for every sender that came back (169
us on this Pi, most of a wake's CPU): asymmetric work in proportion to the
links, which design 011 counts as a reason to design again. The secret is
now remembered in a file beside the key (KEY.links, 0600, begun again for
another node's key): a place of 64 bytes a sender in the order allowed,
mapped, so a sender at rest takes no memory for it. A place is the first
28 bytes of the sender's key, a check (FNV-1a of the key and the secret)
and the secret, written secret first and key last; a place whose check
fails is made again. The UDP link checks see a receiver started again
make no X25519.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
M1 of design 011. A Pure node allows 1,000 to 1,000,000 senders; each
sends one order and, once told it is stored, rests. At rest a link costs
the node 132 bytes (the sender allowed, 73; its number heard, about 60)
and no transport state; with a million the node runs in 139 MB. A quinn
0.11 service holding the same exchange's clients connected at rest costs
38,106 bytes a connection: 289 times as much, past the design's 1/100.
A link at rest takes its next record in 0.56 ms to stored, one X25519 on
the node included; a new QUIC connection takes 1.06 ms and 2.8 times the
receiver's CPU, without storing.
research/experiments/019-idle-links: links.c, many senders written from
serve/link.h's packet format alone (a second implementation), and the
quinn service and clients (quic/, built offline); run.py makes both
measurements.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Each run a listener knew held two OpenSSL contexts. With 16 KiB pages,
what those allocations left behind when idle senders went kept 42 MB of
free heap resident after 10,000 senders had come and gone (in use: 0.6
MB). A run now keeps its keys (16 bytes each way and the nonce bases),
and the listener seals and opens with one context, giving it each
packet's key: about 130 ns a packet on this Pi. Two runs a sender, not
four; a sender's address as an IPv6 one (an IPv4 one fits). What idle
senders held goes back to the system once 1,024 have gone, or 10 s after
any went (malloc_trim). HKDF is fetched once, not at every derivation.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A sender allowed is now a peer of 48 bytes (its key, its key's id, its
sources), found through an index by key id; ids are 32 bits, so keys that
share one are each tried. What a sending peer needs besides (the secret
its key shares with the node's, its runs' keys and addresses, its
streams) is made at its first packet and let go once it has been quiet
for the idle period (--link-idle-ms, 30 s at first) with nothing of it
waiting: the next packet makes it again, one X25519, no handshake. The
listener hands over and answers only the streams and runs that changed,
not every one at each turn. --senders FILE allows senders a line each.
The UDP link checks take their sender from such a file among a thousand
others, let it go after 400 ms of quiet, and see an order after a longer
pause arrive all the same.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
The last number heard from each source was written whole into the links
value at every save, and counted in two bytes: a node that heard from
more than 65,535 sources stored a wrong count, and each save cost as many
bytes as there were sources. The numbers are now a runtime-owned list
{source, seq} in order of source, stored as a list state is (its image
"0heard" and its changes in the log), changed with the event that
delivers, so a failed save takes them back with it. Links stored before
(PLK1) move their numbers to the list at a start and are stored again
without them (PLK2).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pure-serve --send OUTPUT=pure://host:port/INPUT sends an output's saved
sends to another node's --link-port with no handshake: the first packet
carries records. Keys come from the two nodes' keys (X25519, HKDF-SHA256,
AES-128-GCM), with a random run in each packet's header so no nonce
repeats across restarts; nothing is stored for them. What lasts is the
apps' own: the sender's outbox and the receiver's heard numbers
(pure_app_heard, new). The sender detects losses as RFC 9002 says, with
NewReno, and trims a send once the receiver says it stored it; the
receiver hands the owner whole records in order, which it applies from
the one after the source's last heard number, stopping at a gap, and
answers how far it stored with every acknowledgment. Stored bytes are
the bytes on the wire: an output and an input of one shape need no JSON.
server_links_udp kills the receiver, the sender, then both (-9) while
four clients buy: every saved order is applied once, with no handshake
(five runs, 10k-11k orders each). server_links_udp_lossy does it again in
a namespace of its own with the link's packets delayed, lost (3 %) and
reordered (10 %) by netem (three runs, 11k-12k orders).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The first draft replaced a few of QUIC's layers with what Pure already
has, every technique known and the gains constant factors, and it kept
the connection. This one removes it. Nodes have keys, links are the app's
wiring, paths are a cache of where the last authenticated packet came
from, and a link's transport state (key epoch, stored-through number,
heard numbers, packet-number reservation) is a row committed atomically
with the app's state; memory holds only active links and per-path
congestion state. So a restart of either node is invisible to the other
(no handshake, nothing lost or doubled), an idle link costs a row on
disk rather than process memory, and moving the store moves its links.
Forward secrecy comes from key epochs rather than per-connection
handshakes; replay protection survives restarts because heard numbers
are stored with the events they applied (OSCORE, the closest prior art,
needs an Echo round trip there).
Hypotheses H1-H4 and a measurement plan against quinn and ngtcp2:
memory per idle link at 10k to 1M links, kill -9 under load with 10k
active links, moving the store, first contact, and CPU, loss and fairness
as guardrails; stop if an idle link is not at least a hundred times
cheaper than a QUIC connection, if restarts need handshakes, or if
anything is lost or doubled. Beyond it: speculative execution of
verified, deterministic modules, for interactions that do not wait.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Outside feedback 002 asks for a protocol of Pure's own that beats QUIC.
Pure's delivery is already stored, numbered, applied once, and sealed per
link; QUIC and TCP with TLS own the same things again (retransmission
buffers, replay protection, encryption, flow control, acknowledgment).
The design takes the end-to-end argument literally: the sending app's
store is the retransmission buffer, one packet carries both the packet
ACK and the stored-through number, a direct link is sealed once with the
nodes' session keys, first contact carries records in its first packet
(the receiver's key is wired in advance and applying once makes replays
harmless), credits come from what the next save can take, the session is
known by its key rather than its addresses, and stored bytes are the
bytes on the wire. Loss detection, congestion control and cryptography
stay standard (RFC 9002, NewReno then CUBIC, HPKE), plain UDP sockets
rather than SOCK_RAW, HTTP stays the boundary to others.
Written before any code: the decision (this transport, QUIC with Pure's
format, or neither), the open questions, the smallest checks, a
measurement plan against quinn, ngtcp2 and the current HTTP delivery
(seven workloads, RAM and SD stores, loss and delay from unprivileged
namespaces with netem, checked to work on this Pi), and stop criteria:
no transport if it is under 30 % of a delivery's CPU; stop below 1.5x
the best QUIC per CPU; adopt at 2x with a round trip fewer, a quarter of
the memory per idle link, fair to TCP, every correctness and security
check passing, and under 5,000 lines.
|
| | |
|
| | |
|
| |
|
|
| |
and an SD card
|
| | |
|
| | |
|
| |
|
|
| |
store
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
searched without branches
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A fold whose kind has PURE_FOLD_SLICED (128), and a walk whose argument
has PURE_EACH_SLICED (bit 0), first take two numbers, a low and a high
key, and run for the items of their list (keyed by a number) whose keys
are from low up to before high, found by key: the range is the program's
meaning, not an optimization, so their blocks may fault (items out of the
slice never run). A fold a leaf at a time now leaves out the lanes of a
leaf outside the items it runs for, and a leaf that would fault runs item
by item over those items alone. Slices are neither shaped nor given
automatic ranges.
ranges now checks a count over a slice whose test divides (a division by
zero failing it only when inside the slice) against a model, the same on
the item-by-item path, and a walk over a slice.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A filter, a fold or a walk's where (a test at the start of its block and a
jump to its end when it fails) that ANDs comparisons of a keyed list's key
with values reading no item gets a range at load (PureRange: a lower bound
GT or GE, an upper LT or LE, EQ both). When it runs, the bounds are found
once and looked up by key, and only the leaves between the two positions
are read: O(log n + k) where it read every item. A filter leaves the items
out of range out anyway, so it always may; a fold or a walk only when the
rest of its test and its value cannot fault (no division, sum or product
that may overflow, key or item that may be missing), since the items it
skips would have run them; a walk's bounds must also read no state, local
or list its block writes, as the where reads them anew for every item.
A bound that faults leaves the test to run for every item, as written.
A fold of kind 5 (distinct) counts the different values its included
items give (-0 as 0), with a set per fold; it runs a leaf at a time and
is never shaped. max, min and first alone fault over no item.
ranges checks, against the same module reading every item, over 60 lists
of up to 400 keyed items and 3,600 views: a count between two bounds, a sum
between a lower and an upper bound with another condition, first equal to
a key, min with the bound on the right, a count whose test may fault (no
range), a filtered table in bounds, 359 walks changing and removing the
items in bounds, and a walk whose body moves its bound (no range).
lane_folds now makes distinct folds too.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
LOCAL_LOAD and LOCAL_STORE name one of an event's 64 slots, what it holds
(a number, or an index of a list) and that list: slots start at 0 when an
event begins, are shared by its handlers, the walks in them and the folds
in those, and are never saved nor in the states' namespace. A handler uses
a slot for one thing, so its type is verified as a state's is; views have
no locals (IN_LOCALS: a handler's block, a walk in it, a fold in either;
stores only in a handler and its walks). A fold reading a local still runs
a leaf at a time (one value for every item) and a shaped fold takes one
only on its key's side. The signature table's where widens to 16 bits.
locals checks a handler that sums items into a local in a walk, counts in
a fold the items below it, keeps an index in a local and reads it back,
over 80 events against a model, and five refusals. server_rows' shop now
takes stock through an index local and counts in one.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
GET /views/name?argument=value gives a view that takes arguments its
arguments: the query as it came is kept on the request's work slot (it
outlives the connection, which may go before the view is shown), and when
the view is shown, on the connection thread from a snapshot or by the
owner, its pairs (values %XX decoded) are laid out as a payload of the
arguments' shape and given before the view is evaluated. from and count
still page a table. An argument missing, unknown, given twice, not of its
type or badly escaped answers 400 invalid. A registered query takes its
view's arguments the same way (GET /queries/name?k=4). An arguments input
is not an input to post to.
server_arguments opens 200 accounts, then asks 83 times against a model:
balances by key (404 no_value for a key not there), counts above an
argument, texts with &, =, % and non-ASCII shown back, a table filtered by
two arguments whose column folds with one, paged, counts by an owner's
name, a registered query with an argument, and six refusals. The format's
document (008) describes arguments.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
An input of the new kind 6 holds the arguments a view takes: fields as a
message has, no handler, the view's number where a key input names its
key, and the view's name with .args after it; one set a view, never a
column's (a table's columns and filter read the table's). The view, its
columns and filter, and the folds in them read their arguments with
FIELD, verified as a handler reads its message (IN_ARGUMENTS: reads, no
reply or document), and a fold anywhere may now read the message or
arguments of the block it is in: a handler's count(a in accounts where
a.bid == pay.bid) verifies.
pure_app_arguments gives a view's arguments as a payload of their shape
(placed as a message's fields); the view shows nothing until they are
given, and a filtered table keeps its rows for the arguments they were
made for. A text argument a view shows stays in the caller's payload
(FROM_ARGUMENTS), so it outlives the evaluation. A fold reading a number
argument still runs a leaf at a time (the same value for every item), and
a shaped fold takes arguments only on its key's side, so its pass does not
depend on them. Message fields are placed by one function for messages
and arguments.
view_arguments checks a keyed lookup by argument, a count above an
argument for 43 values against a model, a text argument shown back as it
is, a table filtered by an argument whose column folds with it (rows kept
per arguments), a handler's fold over its message, and five refusals.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A rows field's JSON value is an array of objects naming the record's
fields (numbers, strings; a field left out takes its default). The
connection thread encodes them as the list's stored form, which may be
longer than the JSON, grows the request's input by that much
(pure_runtime_grow: charged to the payload budget as admission charges an
input, 503 payload_budget when it does not fit) and places them after the
body, so the owner still applies a message it need not parse. A message
whose only variable field is rows takes a body that is not JSON as those
rows in their stored form: a bulk insert without JSON. An API's answer
(--reply) gives rows the same way. A delivered batch holds no rows.
server_rows loads 30,000 stock rows in one JSON request (some 250,000
rows a second, against 6,500 with a request a row) and 20,000 more as
bytes, refuses a load with a key already there whole (409 exists), then
checks 400 random orders against a model (294 applied, 106 refused with
none of their rows applied), texts with escapes and Korean, a walk that
removes, and a restart that keeps every state and no rows.
The format's document (008) describes walks, rows fields and lists of
rows, and OUTER_ROW.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
EACH runs the block that follows in a handler once for every item its list
has when it begins, in order, with ROW the item: the block may change
states, the list's items (a field, or removing one) and other lists, but
may not insert into the list it walks, hold another walk or reply, so a
walk runs at most its items times its block. Removing the walked item, or
one before it, moves the walk with it, so no item is skipped. A fold in
the block reads the walked item as OUTER_ROW, as a fold in a column reads
its row. SQL's UPDATE ... WHERE and DELETE ... WHERE are a walk.
A message field may now be rows (type 5, naming a list state): records
laid out as the list's stored form, filling that list, whose flags hold
PURE_LIST_ROWS and whose name is the field's path (order.lines), when the
message arrives; rows that are not whole fail it (limit). A list of rows
is emptied when every event ends, is never saved, and its changes are not
logged for undoing, so a handler walks, folds and counts a message's rows
as any list, and may use one as a scratch table within its event. An
output has no rows. pure_list_clear lets a list's items go.
walks checks a shop against a model over 3,000 random events: orders
whose rows each take stock and are logged, all or nothing (997 applied,
1,322 failed and undone, by an item not stocked or not enough of it), and
a clean that walks the stock removing items (83 while walking, 32 of them
before the walked item). Nine refusals are checked, and 20,000 damaged
walk modules run safely with no rows left after any event; all under
ASan and UBSan too.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
091 measures four queries on pgbench's tables of 100,000 accounts, the
same rows and answers on Pure (registered queries, HTTP and JSON
included) and SQLite 3.46.1, through each step: items field by field in
leaves, folds and filters a leaf at a time, and one pass answering a fold
in a column for every row. Per-branch totals went from 398 ms to 4.9 ms
(SQLite 45, 14.3 with a DBA's indexes), a three-table join from 3.3 to
0.48 ms (1.47), and each rich account's rank from 9.2 s to 6.8 ms
(2.84 s, 7.6 ms with an index); counting accounts in credit takes 1.05 ms
where an index lets SQLite count 0.16 ms. Pure declares no index.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A fold in a table's column runs once per row, over its whole list each
time: per branch, the sum of its accounts read all 100,000 accounts for
every branch, and per rich account, how many are richer read them all for
each of 404 accounts. At load, a planned fold in a column whose test
compares a field of its item with a key the row gives, the rest of it
reading only its list, its item and constants, gets a shape: a group (the
field equals the key) or a rank (a count of items whose field orders
against the key, and nothing else). The app then keeps one pass over the
list per fold and list version: a group pass runs the block a leaf at a
time with the comparison taken as true and the key left out, folding each
item into its field's group; a rank pass radix-sorts the field's values.
A row runs its key alone and looks it up. The pass is made at the second
row that asks, so a fold asked once scans once.
Answers are the fold's own: items join their group in order, so a sum
adds as it would; a key that faults leaves that row to run the fold as
written, and an item that would fault in the pass leaves every row to.
lane_folds now also makes shaped folds (185 groups, 127 ranks, 98 passes
made) and checks them against the same module run item by item.
On the bank of 100,000 accounts: per-branch totals 4.8 ms (16 ms before,
SQLite 46 ms, 14 ms with the index a DBA would add on branch and balance)
and each rich account's rank 6.7 ms (268 ms before, SQLite 2.84 s, 7.6 ms
with an index on balance), HTTP and JSON included on Pure's side. Pure is
given no index.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A fold ran its block once per item through the interpreter, a call and a
lookup of the item each: counting 100,000 accounts took 4.7 ms and a
per-account rank over them 5.3 s. When every instruction of a fold's
block (or a table's filter) has a lane form, as numbers and indexes with
no branch and no text do, the module marks it at load, and the block
then runs for a leaf's items at once: a value is one per item (a leaf's
column read in place, or computed into a row per stack depth), the
items' positions, or one value for all. Lookups by a value for all items
are made once per fold, not per leaf. Leaves are walked with an iterator
that keeps its path (pure_list_next) and fetches the next leaf ahead.
The lanes only take what the interpreter would compute. Where an item
would fault (division by zero, a non-finite result, a key not found, an
index past the end, a clamp the wrong way round, a fractional move), the
whole leaf runs again item by item, so a fold faults, and a filter leaves
the item out, exactly as before; items join the fold in order, so a sum
adds as it did. lane_folds checks it: 300 rounds of random well-typed
folds in views, in a table's columns reading their row and in a handler
storing them, with a random filter, over random lists, answer bit for bit
as the same module run item by item, 920 faults included by kind and
instruction. list_trees now walks runs too.
On the bank of 100,000 accounts (benchmarks/bank/queries.py), against
the same SQL on SQLite 3.46.1: per-branch totals 16 ms (SQLite 46 ms),
accounts in credit 1.1 ms (6.2 ms), large transfers with their accounts
and tellers 0.77 ms (2.1 ms), each rich account's rank 268 ms (2.85 s),
HTTP and JSON included on Pure's side.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
An item was its own allocation (a share stamp and a cell per field) that
a leaf pointed to, so reading a field of items one after another chased a
pointer to wherever each was made: a count over 100,000 accounts built by
HTTP took 14.5 ms, against 4.6 ms after a restart laid them out in order.
A leaf now holds its items' cells field by field, numbers as binary64
arrays and texts as spans, and pure_list_run gives a leaf's columns to
read item after item.
Copy on write moves to the leaf: a change copies the path including its
leaf, whose texts stay shared; texts carry the share stamp items had, so
one a snapshot may read is retired, not freed. An event's log keeps a
removed item's cells until the event is kept or undone. A node split at
the end of the list leaves the full node full, so lists that grow at
their end (histories, keyed lists filled in key order) fill their leaves.
An account of three numbers and a text now takes 41 bytes in a leaf, not
some 130 across its allocation and the leaf's pointer.
The same count now takes 4.7 ms however the list was built, below
SQLite's 6.1 ms on the same rows. list_trees passes unchanged, and the
app model's tests under ASan and UBSan.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
PUT /queries/NAME takes a module compiled with aup query: the node loads
and verifies it, checks that its states are the app's (names, types,
records) and that it has a view NAME, and keeps it (64 at most; 409 for
another app's module, 400 for bytes that are no module, 404 without the
view). GET /queries/NAME?from=&count= shows that view from the last
save's snapshot with a reader of the query's module, as a declared view
is shown, so a query runs beside writes and sees one save whole. A query
registered again answers anew.
server_queries registers a per-branch total (a correlated fold), a join
of history rows with their accounts and tellers, each rich account's
rank and a filtered count, checks them against a model of the same
transfers, asks while transfers go on, pages, and checks the refusals.
|
| |
|
|
|
|
|
|
|
|
| |
OUTER_ROW pushes the row of the column a fold's block runs for, as an
index of the column's list; ROW inside the block stays the fold's own
item. The verifier accepts it only in a fold's block inside a column,
naming that column's list (a fold in a handler, a plain view or a filter
has no outer row). So a column can hold a correlated fold: per branch,
the sum of its accounts; per account, how many are richer. aup compiles
the column's row inside a fold to it.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
benchmarks/bank/bank.aup keeps accounts, tellers and branches as keyed
lists and a history list; a transfer changes a row of each and appends a
history row in one event, as pgbench's TPC-B-like transaction does in
SQL. run.py makes a scale's rows and drives transfers through the load
generator's new transfer kind; sqlite-bank.c runs the same tables and
SQL on SQLite 3.46.1, a connection a thread, WAL.
With lists as trees, transfers run at 30.5k a second on 1,000 accounts
and 24.5k on 100,000 (RAM store, one connection thread, HTTP and JSON
included), where they ran at 3.2k on 30,000 before. Durable on the SD
card, Pure makes 1,339 a second and SQLite with synchronous=FULL 84 to
153. SQLite in process with no HTTP makes about 40k in RAM: recorded as
it is, with the write amplification of 100,000-row tables on the card
and ad hoc queries as what comes next.
|
| |
|
|
|
|
|
|
|
| |
A list's stored form is one value in the tree store, so the limit on a
document (1.38 MiB, what a host carries whole as an HTTP body) capped a
table at about 51,000 rows of 28 bytes. A value the runtime owns, a state
(a key starting with '.') or a draft without a key, may now grow to 1 GiB;
documents keep their limit. The store's format is unchanged: its trees
already reach far past either.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A list was one array of item pointers: an item's place in the stored form
was found by adding up the sizes of every item before it, so each change
and each appended row cost O(n); a snapshot copied a changed list's whole
array at each save; and a list held at most 65,536 items. A bank of
30,000 accounts ran transfers at 299 us of CPU each, and set up in time
growing with the square of its rows.
Items now live in a B+tree of fanout 32 whose nodes know the items below
them, their stored bytes and the first of them: positions, keys (keyed
lists) and stored offsets are found in O(log n). Nodes carry the list's
share count when made, as items do: one a snapshot may read never changes,
so a change copies the path to it and retires the nodes it replaced, and
a share is a root and a count. Reads in order go through a finger, each
thread's last leaf. A list may hold 2^24 items.
list_trees checks it against a model array: 6,000 events of inserts,
removals and field changes, 1,195 of them undone, up to 8,983 items (three
levels), the stored form equal byte for byte, four shares kept whole while
the list changes on, and 13,959 keyed items found by key as a binary
search finds them. The app model's tests pass under ASan and UBSan too.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The roadmap's E2EE messenger made of the runtime's general parts, with
no messenger code in it: sealed, authenticated sessions between nodes
(086, 088), Vega on each node's display as a surface (design 010), the
app in Vega's examples. Vega's check runs two nodes through a recording
relay: a line typed on one display is drawn on the other, frames match
the vega command's, and the relay sees no text.
A line takes 0.79 ms at p50 and 1.1 ms at p99 from one node's program
to the other node's views with stores in RAM, both commits, the HTTP
delivery and the seal included (sealing adds nothing measurable); on the
Pi's SD card 3.5 ms at p50, the long tail being the card's syncs.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pure-serve --surface LIBRARY [-- ARGUMENTS] maps a library and runs its
pure_surface_run in a thread of its own (serve/surface.h). The surface
reads the app as of the last save where it is, a published snapshot in a
reader app, as a connection thread reads views; an eventfd tells it of
each new snapshot; and its events (a key pressed, text typed) are
requests without a host in the owner's queue, applied and saved with the
round, their results handed back to the surface's waiter. So one node
serves programs over HTTP and people on its display with one owner of
its states, and nothing between the states and the screen. pure-serve
exports its pure_* functions for the surface to call, one copy of the
app model in the process; the table carries sizeof(PureApp) and
sizeof(PureModule) so a surface built for another layout refuses.
The check builds a probe surface that types, presses and reads: its
events show in HTTP views, an HTTP input shows in it, wrong inputs are
refused, it stops with the server, and a module without states cannot
have one. Without a surface the request path measured the same (small
GETs 140.9k vs 139.9k, app totals 126.3k vs 126.3k over eight rounds);
the exported symbols cost 11 KB of the file and 24 KiB idle.
|
