aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
...
* Wait for late packets, undo what they cost, and grow as CUBICeunhhu2026-09-294-13/+311
| | | | | | | | | | | | | | | A packet called lost stays known a while; acknowledged after all, it came late: what it carried counts as received, the reordering awaited grows (later packets acknowledged, and time past the RTT, as RACK), and an episode of congestion events whose losses all came late is undone. The window grows only when it was used (RFC 9002 7.8), and congestion avoidance is CUBIC (RFC 9438), as quinn's. In a path of 50 Mbit/s whose packets reorder by up to 5 ms, with no loss, a link carried 1.5 Mbit/s at an RTT of 20 ms and 10 at 100 ms; now 43.6 and 40.1, as without reordering (research/experiments/020-lossy-paths, virtual time). With 1% random loss: 4.9 to 6.1 at 20 ms, 1.0 to 1.3 at 100 ms. The transport's CPU a record is unchanged.
* Say why the links became a part, and not a repository (009, 011)eunhhu2026-09-292-3/+38
|
* Make the load generator's devices real link senders, many on one threadeunhhu2026-09-294-384/+170
| | | | | | | | | | links.c was a second implementation of the packet format; each device is now a PureLinkSender on a socket of its own, keyed with a secret made outside (pure_link_sender_shared: a key kept elsewhere, or a test's many). M1 (1k, 10k) and M2 (10k, kill -9) pass with it: no silent step, caught up in 0.2 s, every order applied once; the slowest store after the restart is now the sender's 1 s wait for a word of the save (STORED_WAIT), where the old devices tried again after 200 ms.
* Keep a link's two sides as state alone, the sockets and threads aparteunhhu2026-09-298-1555/+2054
| | | | | | | | | | | | link/link-protocol.h is the protocol with no socket, thread or clock of its own: packets and time in, packets out. link/link.h carries it over UDP as before. A sender starts small (8 flights, 16 records, a 4 KiB ring) so one thread can hold many. The transport's CPU a record is the same within 2% (five rounds each way, 61 B to 16 KiB). test-link runs both sides through a network of its own: loss, packets held back and doubled, failed saves, an idle sweep, and both sides starting again, every record applied once as sent.
* Let links know nothing of the core: the host names inputs and shapeseunhhu2026-09-297-105/+137
| | | | | | A sender takes its messages' shape, and a listener asks its host which input a stream goes to. The core gives the shape (pure_module_shape); link/ now builds without the core's headers.
* Put the links and the crypto they use in link/, a part of their owneunhhu2026-09-2916-35/+47
| | | | | libpure-link.a holds them; pure-serve and the M5 harness link it instead of naming its source files.
* Measure the drains again with the outbox as a list (096)eunhhu2026-09-292-17/+48
| | | | | | | Between two pure-serve nodes, 16 KiB records drain at 33,600 a second (23,000 before), the links spending 58 us a record (76), under quinn's 60 while the receiver also stores each one. 1,013 bytes: 406,000 a second (347,000). Saving sends one by one: 5,000 of 16 KiB in 0.91 s (84.8).
* Keep the outbox as a list: a save logs only the sends and trims it madeeunhhu2026-09-297-153/+277
| | | | | | | | | | | | | | | | | | | | | The sends not yet trimmed were written whole into the links value at every save, as the numbers heard were: a sender whose receiver is away or slow saved all its waiting sends again with each new one. They are now a runtime list {output (its name), seq, payload}, in the order sent, stored as a list state is (image "0sent" and its changes in the log). A send is a change of the list with its event; a trim lets the sends go at once and a failed save brings them back; the host sees the saved ones (sent_saved). Outputs are named in the items, so a module replaced keeps them without moving anything; one that lacks or changed an output with sends left is refused as before. Links stored before (PLK1, PLK2) move their sends to the list at a start and are stored again without them (PLK3). A list field may now hold any bytes (kind 2), unchecked, besides a number and a UTF-8 text. Saving 5,000 sends of 16 KB one by one took 84.8 s and left a store of 400 MB; now 0.91 s and 144 MB. 20,000 of 61 bytes: 8.4 s, now 2.9 s.
* Measure the CPU a record costs a link, against quinn (096)eunhhu2026-09-296-22/+395
| | | | | | | | | | | | | | | | | | | | | M5 of design 011. At first a link spent eight times quinn's CPU on a small record: scans of the backlog, a system call and allocations a record, 19 bytes of framing. After the seven changes before this one, the transport alone (transport.c; a backlog drained, every record checked) spends per record 0.30 us for 61 bytes (quinn 0.27: 1.06 to 1.12 times over three campaigns, at the design's 10 % line), 2.91 us for 1,013 bytes (quinn 3.88) and 45.2 us for 16,013 bytes (quinn 60.9): 25 % less for the larger ones, and 30 % more records a second. What is left for small records is per packet: the stream's names and the RECORDS head. quinn's numbers carry no confirmation of storing and no exactly once. Between two pure-serve nodes the links spend more per record, woken at the receiving app's pace. Filling the sender showed the outbox is written whole at every save, as the numbers heard were (094): the next thing to change. run.py gains --transport and --drain; the quinn service reads a drain a chunk at a time and its client writes 64 KiB at a time.
* Send a record whole when it fits a packet, however many pieces it haseunhhu2026-09-293-46/+75
| | | | | | | | | | | | | | | | | A record over a piece (1,000 bytes) went in pieces even when it fit the packet whole: a record of 1,013 bytes went as two RECORD frames, and the receiver gathered it the slow way (an allocation, pieces, then onto the line). A record none of whose pieces went yet now goes whole in a RECORDS frame whenever it fits what is left of the packet; its pieces are marked sent together. A record too large for a packet, or one resent after some of its pieces were lost, still goes in pieces, and a receiver takes a record whole or in pieces alike. The transport alone takes 2.85 us a record of 1,013 bytes, not 3.18 (quinn: 3.90). transport.c now checks every record delivered: in order, once each, its bytes as sent. None was wrong in a network namespace with 1 ms of delay, 3 % loss and 10 % reordering, with records of 61, 1,013 and 16,013 bytes.
* Keep a link's queued records in one byte ring, taken back as they are storedeunhhu2026-09-281-23/+65
| | | | | | | | | | | | | A queued record had an allocation of its own for its bytes and its pieces' states, freed when the receiver stored it. Records come and go in order, so their bytes now go one after another into a ring, never cut by its end, and go back as the ring's head moves past the ones stored; a ring too small grows to twice its size, the records moved into it in order. A record of one piece keeps its piece's state in itself. The transport alone drains 200,000 records of 61 bytes at 5.3 to 6.5 million a second; per record the link takes 0.137 us, the listener 0.14: 0.28 in all, against quinn's 0.26 a message over one stream.
* Keep a stream's records in order on a line, handed over as they areeunhhu2026-09-281-13/+93
| | | | | | | | | | | | | | A receiver copied each record into an allocation of its own, copied them all again into a delivery, and freed each once stored: about 110 ns a record of 61 bytes. Once the last number heard is known, whole records that come in order now go onto the stream's line as a delivery carries them (u32 size, bytes). The line is handed over as it is, and let go by moving its head; its front is given back when there is no room at its end. Records that come early or in pieces, or before the last heard is known, still wait in held, and move onto the line once it reaches them. The transport alone: the listener took 0.20 us a record of 61 bytes, now 0.15; with the link, 0.31 us a record in all, against quinn's 0.26.
* Take a burst whole with UDP GRO; key the listener's AES only when the run ↵eunhhu2026-09-281-11/+63
| | | | | | | | | | | | | | | changes A listener's socket now asks for GRO: the kernel hands over the datagrams of one sender that came one after another as one buffer, each the segment's size but the last, and on the loopback a burst sent with GSO comes whole instead of cut into datagrams. The listener takes up to eight such buffers at once. It gives its AES-GCM context a key only when the packet's run has another than the one it holds, for opening and for sealing answers. The transport alone drained 200,000 records of 61 bytes at 4.2 million a second (0.454 us a record, both sides); now at 4.9 million (0.394 us).
* Send small records in a RECORDS frame: 1 or 2 bytes of framing each, not 19eunhhu2026-09-282-9/+74
| | | | | | | | | | | | | | A RECORD frame carries a number, a size, an offset and a length before each record's bytes, 19 bytes: a packet held 13.7 records of 61 bytes. A RECORDS frame (0x06) carries the first record's number and a count, then whole records whose numbers follow on, each its size as a varint and its bytes: 17.2 a packet. A piece of a record larger than a piece still goes in a RECORD frame; a receiver takes both. The transport alone (research/experiments/019-idle-links/transport.c) drains 200,000 records of 61 bytes in 11,600 packets instead of 14,600; per record the link takes 0.217 us and the listener 0.23, 0.447 us in all, against quinn's 0.43 a message over one stream.
* Wake a link's thread only when it sleeps, onceeunhhu2026-09-282-3/+178
| | | | | | | | | | | | | | | Every queued record wrote the link's eventfd, a system call each, and woke the thread to send a packet of a few records. A record queued now wakes the thread only when it sleeps and nobody woke it yet; one queued while it sends marks the link, and the thread looks again before it sleeps. With records queued one at a time as fast as a thread can, 200,000 small ones went at 400k a second, and now at 730k; queuing one costs 0.43 us, not 1.0. research/experiments/019-idle-links/transport.c measures the transport alone: a link and a listener in one process, no app, no store. A backlog of 200,000 records of 61 bytes drains at 3 to 4 million a second, 0.24 us a record on each side.
* Drain a link's backlog without scanning it: 370k to 990k records a secondeunhhu2026-09-281-105/+218
| | | | | | | | | | | | | | | | | | | | | | A backlog made the link slow in its square: the sender looked at every queued record's pieces twice a turn to know whether any waited to be stored, and the receiver moved all its held records down each time it let the first one go. - The sender counts the pieces received and not stored, and keeps where the unsent ones begin; every change of a piece's state goes through one place that keeps both. - The receiver keeps its held records from a head index. One in order goes last, and one out of order finds its place by halving. The record's bytes and its pieces' marks are one allocation. - Packets go in bursts with UDP GSO, 54 at a time: 0.45 us a packet on this Pi, against 2.4 us for a send each. A full packet is padded to a whole datagram (PADDING frames, as QUIC does). Datagrams are taken 64 at a time (recvmmsg). Twenty thousand saved sends of 61 bytes drained at 370k a second; now at 990k. Per record, the sender's link thread took 1.1 us and now takes 0.5; the receiver's took 2.4 and now takes 0.5.
* Measure a restart under load and a first contact (095)eunhhu2026-09-289-30/+341
| | | | | | | | | | | | | | | | | | M2 and M4 of design 011. Ten thousand devices send an order every five seconds (2,000 a second); the receiver is killed (-9) at about 21 s and started again at once. With its secrets remembered the node shows no 100 ms step without an order stored, catches up in 0.2 s, spends no more CPU than before and makes no X25519; 34 orders were sent again, and every order made was applied once (three rounds; on the SD card the same, with 0.2 s silent). Without the memo it makes 10,000 X25519, 1.6 s of CPU. quinn under the same load stops for 1.0 s and catches up after 10.9 s: 10,115 new connections, about 7 s more of the service's CPU and 8 s of the clients'. A first contact takes Pure one round trip and QUIC two, at 10, 50 and 100 ms of RTT. run.py --restart and --first-contact make them; the quinn harness gains a fixed port with a reset key from its key, clients that connect again, and a drain for M5.
* Name the link threads, so their CPU is seen apart (pure-link, pure-listen)eunhhu2026-09-281-0/+2
|
* Remember each sender's secret beside the node's key: no X25519 at restarteunhhu2026-09-284-10/+89
| | | | | | | | | | | | | | A listener lets a quiet sender's secret go with its active state, so a node started again made one X25519 for every sender that came back (169 us on this Pi, most of a wake's CPU): asymmetric work in proportion to the links, which design 011 counts as a reason to design again. The secret is now remembered in a file beside the key (KEY.links, 0600, begun again for another node's key): a place of 64 bytes a sender in the order allowed, mapped, so a sender at rest takes no memory for it. A place is the first 28 bytes of the sender's key, a check (FNV-1a of the key and the secret) and the secret, written secret first and key last; a place whose check fails is made again. The UDP link checks see a receiver started again make no X25519.
* Measure a link at rest: 132 bytes against a QUIC connection's 38,106 (094)eunhhu2026-09-2810-1/+2123
| | | | | | | | | | | | | | | | | M1 of design 011. A Pure node allows 1,000 to 1,000,000 senders; each sends one order and, once told it is stored, rests. At rest a link costs the node 132 bytes (the sender allowed, 73; its number heard, about 60) and no transport state; with a million the node runs in 139 MB. A quinn 0.11 service holding the same exchange's clients connected at rest costs 38,106 bytes a connection: 289 times as much, past the design's 1/100. A link at rest takes its next record in 0.56 ms to stored, one X25519 on the node included; a new QUIC connection takes 1.06 ms and 2.8 times the receiver's CPU, without storing. research/experiments/019-idle-links: links.c, many senders written from serve/link.h's packet format alone (a second implementation), and the quinn service and clients (quic/, built offline); run.py makes both measurements.
* Keep a run's keys as bytes: one AES-GCM context serves the listenereunhhu2026-09-285-66/+100
| | | | | | | | | | | | Each run a listener knew held two OpenSSL contexts. With 16 KiB pages, what those allocations left behind when idle senders went kept 42 MB of free heap resident after 10,000 senders had come and gone (in use: 0.6 MB). A run now keeps its keys (16 bytes each way and the nonce bases), and the listener seals and opens with one context, giving it each packet's key: about 130 ns a packet on this Pi. Two runs a sender, not four; a sender's address as an IPv6 one (an IPv4 one fits). What idle senders held goes back to the system once 1,024 have gone, or 10 s after any went (malloc_trim). HKDF is fetched once, not at every derivation.
* Let a link at rest cost the listener only its allowingeunhhu2026-09-284-172/+433
| | | | | | | | | | | | | | | | A sender allowed is now a peer of 48 bytes (its key, its key's id, its sources), found through an index by key id; ids are 32 bits, so keys that share one are each tried. What a sending peer needs besides (the secret its key shares with the node's, its runs' keys and addresses, its streams) is made at its first packet and let go once it has been quiet for the idle period (--link-idle-ms, 30 s at first) with nothing of it waiting: the next packet makes it again, one X25519, no handshake. The listener hands over and answers only the streams and runs that changed, not every one at each turn. --senders FILE allows senders a line each. The UDP link checks take their sender from such a file among a thousand others, let it go after 400 ms of quiet, and see an order after a longer pause arrive all the same.
* Keep the numbers heard as a list: a save logs only the ones it changedeunhhu2026-09-285-123/+290
| | | | | | | | | | | | The last number heard from each source was written whole into the links value at every save, and counted in two bytes: a node that heard from more than 65,535 sources stored a wrong count, and each save cost as many bytes as there were sources. The numbers are now a runtime-owned list {source, seq} in order of source, stored as a list state is (its image "0heard" and its changes in the log), changed with the event that delivers, so a failed save takes them back with it. Links stored before (PLK1) move their numbers to the list at a start and are stored again without them (PLK2).
* Link two nodes without a connection over UDP (093, design 011's first stage)eunhhu2026-09-2813-26/+1890
| | | | | | | | | | | | | | | | | | | | | pure-serve --send OUTPUT=pure://host:port/INPUT sends an output's saved sends to another node's --link-port with no handshake: the first packet carries records. Keys come from the two nodes' keys (X25519, HKDF-SHA256, AES-128-GCM), with a random run in each packet's header so no nonce repeats across restarts; nothing is stored for them. What lasts is the apps' own: the sender's outbox and the receiver's heard numbers (pure_app_heard, new). The sender detects losses as RFC 9002 says, with NewReno, and trims a send once the receiver says it stored it; the receiver hands the owner whole records in order, which it applies from the one after the source's last heard number, stopping at a gap, and answers how far it stored with every acknowledgment. Stored bytes are the bytes on the wire: an output and an input of one shape need no JSON. server_links_udp kills the receiver, the sender, then both (-9) while four clients buy: every saved order is applied once, with no handshake (five runs, 10k-11k orders each). server_links_udp_lossy does it again in a namespace of its own with the link's packets delayed, lost (3 %) and reordered (10 %) by netem (three runs, 11k-12k orders).
* Redesign 011: no connections, the transport's state as the app's dataeunhhu2026-09-282-151/+172
| | | | | | | | | | | | | | | | | | | | | | | | | The first draft replaced a few of QUIC's layers with what Pure already has, every technique known and the gains constant factors, and it kept the connection. This one removes it. Nodes have keys, links are the app's wiring, paths are a cache of where the last authenticated packet came from, and a link's transport state (key epoch, stored-through number, heard numbers, packet-number reservation) is a row committed atomically with the app's state; memory holds only active links and per-path congestion state. So a restart of either node is invisible to the other (no handshake, nothing lost or doubled), an idle link costs a row on disk rather than process memory, and moving the store moves its links. Forward secrecy comes from key epochs rather than per-connection handshakes; replay protection survives restarts because heard numbers are stored with the events they applied (OSCORE, the closest prior art, needs an Echo round trip there). Hypotheses H1-H4 and a measurement plan against quinn and ngtcp2: memory per idle link at 10k to 1M links, kill -9 under load with 10k active links, moving the store, first contact, and CPU, loss and fairness as guardrails; stop if an idle link is not at least a hundred times cheaper than a QUIC connection, if restarts need handshakes, or if anything is lost or doubled. Beyond it: speculative execution of verified, deterministic modules, for interactions that do not wait.
* Design the Pure transport: stored records sent straight over UDP (011)eunhhu2026-09-282-0/+202
| | | | | | | | | | | | | | | | | | | | | | | | | | | Outside feedback 002 asks for a protocol of Pure's own that beats QUIC. Pure's delivery is already stored, numbered, applied once, and sealed per link; QUIC and TCP with TLS own the same things again (retransmission buffers, replay protection, encryption, flow control, acknowledgment). The design takes the end-to-end argument literally: the sending app's store is the retransmission buffer, one packet carries both the packet ACK and the stored-through number, a direct link is sealed once with the nodes' session keys, first contact carries records in its first packet (the receiver's key is wired in advance and applying once makes replays harmless), credits come from what the next save can take, the session is known by its key rather than its addresses, and stored bytes are the bytes on the wire. Loss detection, congestion control and cryptography stay standard (RFC 9002, NewReno then CUBIC, HPKE), plain UDP sockets rather than SOCK_RAW, HTTP stays the boundary to others. Written before any code: the decision (this transport, QUIC with Pure's format, or neither), the open questions, the smallest checks, a measurement plan against quinn, ngtcp2 and the current HTTP delivery (seven workloads, RAM and SD stores, loss and delay from unprivileged namespaces with netem, checked to work on this Pi), and stop criteria: no transport if it is under 30 % of a delivery's CPU; stop below 1.5x the best QUIC per CPU; adopt at 2x with a round trip fewer, a quarter of the memory per idle link, fair to TCP, every correctness and security check passing, and under 5,000 lines.
* Measure TPC-C again with compact log recordseunhhu2026-09-285-34/+39
|
* Log list changes compact: varint positions and integers, the rest bit for biteunhhu2026-09-284-32/+131
|
* Check TPC-C's consistency from one snapshot a side, and measure TPC-C on RAM ↵eunhhu2026-09-286-24/+235
| | | | and an SD card
* Write list images a chunk at a time, and keep the log within the lists' sizeeunhhu2026-09-286-37/+116
|
* Keep lists as images and a log of their changes, checkpointed off the ownereunhhu2026-09-2810-188/+662
|
* Log an app's saves in reused segments and write list images whole beside the ↵eunhhu2026-09-285-4/+862
| | | | store
* Share the store's digest between its fileseunhhu2026-09-283-71/+81
|
* Gather returning requests only as long as the last save waited for its deviceeunhhu2026-09-281-3/+18
|
* Read and write numbers exactly without strtod or printf in the common caseseunhhu2026-09-285-2/+172
|
* Guide a list's inner nodes: items before each child and its first key, ↵eunhhu2026-09-284-44/+197
| | | | searched without branches
* Save a list as its items and a journal of its changes sinceeunhhu2026-09-288-186/+358
|
* Run TPC-C on Pure and on SQLite, and compare the states they leaveeunhhu2026-09-285-0/+1316
|
* Allow 16 records and 128 record fields a moduleeunhhu2026-09-282-2/+2
|
* Slice a keyed list: fold and walk over a range of keyseunhhu2026-09-284-29/+87
| | | | | | | | | | | | | | | | A fold whose kind has PURE_FOLD_SLICED (128), and a walk whose argument has PURE_EACH_SLICED (bit 0), first take two numbers, a low and a high key, and run for the items of their list (keyed by a number) whose keys are from low up to before high, found by key: the range is the program's meaning, not an optimization, so their blocks may fault (items out of the slice never run). A fold a leaf at a time now leaves out the lanes of a leaf outside the items it runs for, and a leaf that would fault runs item by item over those items alone. Slices are neither shaped nor given automatic ranges. ranges now checks a count over a slice whose test divides (a division by zero failing it only when inside the slice) against a model, the same on the item-by-item path, and a walk over a slice.
* Read only the keys a test allows, and count distinct valueseunhhu2026-09-284-35/+410
| | | | | | | | | | | | | | | | | | | | | | | | | | | A filter, a fold or a walk's where (a test at the start of its block and a jump to its end when it fails) that ANDs comparisons of a keyed list's key with values reading no item gets a range at load (PureRange: a lower bound GT or GE, an upper LT or LE, EQ both). When it runs, the bounds are found once and looked up by key, and only the leaves between the two positions are read: O(log n + k) where it read every item. A filter leaves the items out of range out anyway, so it always may; a fold or a walk only when the rest of its test and its value cannot fault (no division, sum or product that may overflow, key or item that may be missing), since the items it skips would have run them; a walk's bounds must also read no state, local or list its block writes, as the where reads them anew for every item. A bound that faults leaves the test to run for every item, as written. A fold of kind 5 (distinct) counts the different values its included items give (-0 as 0), with a set per fold; it runs a leaf at a time and is never shaped. max, min and first alone fault over no item. ranges checks, against the same module reading every item, over 60 lists of up to 400 keyed items and 3,600 views: a count between two bounds, a sum between a lower and an upper bound with another condition, first equal to a key, min with the bound on the right, a count whose test may fault (no range), a filtered table in bounds, 359 walks changing and removing the items in bounds, and a walk whose body moves its bound (no range). lane_folds now makes distinct folds too.
* Give an event localseunhhu2026-09-285-20/+156
| | | | | | | | | | | | | | | | | LOCAL_LOAD and LOCAL_STORE name one of an event's 64 slots, what it holds (a number, or an index of a list) and that list: slots start at 0 when an event begins, are shared by its handlers, the walks in them and the folds in those, and are never saved nor in the states' namespace. A handler uses a slot for one thing, so its type is verified as a state's is; views have no locals (IN_LOCALS: a handler's block, a walk in it, a fold in either; stores only in a handler and its walks). A fold reading a local still runs a leaf at a time (one value for every item) and a shaped fold takes one only on its key's side. The signature table's where widens to 16 bits. locals checks a handler that sums items into a local in a walk, counts in a fold the items below it, keeps an index in a local and reads it back, over 80 events against a model, and five refusals. server_rows' shop now takes stock through an index local and counts in one.
* Take a view's arguments from the request's queryeunhhu2026-09-283-14/+230
| | | | | | | | | | | | | | | | | | | | GET /views/name?argument=value gives a view that takes arguments its arguments: the query as it came is kept on the request's work slot (it outlives the connection, which may go before the view is shown), and when the view is shown, on the connection thread from a snapshot or by the owner, its pairs (values %XX decoded) are laid out as a payload of the arguments' shape and given before the view is evaluated. from and count still page a table. An argument missing, unknown, given twice, not of its type or badly escaped answers 400 invalid. A registered query takes its view's arguments the same way (GET /queries/name?k=4). An arguments input is not an input to post to. server_arguments opens 200 accounts, then asks 83 times against a model: balances by key (404 no_value for a key not there), counts above an argument, texts with &, =, % and non-ASCII shown back, a table filtered by two arguments whose column folds with one, paged, counts by an owner's name, a registered query with an argument, and six refusals. The format's document (008) describes arguments.
* Let a view take argumentseunhhu2026-09-283-55/+289
| | | | | | | | | | | | | | | | | | | | | | | | | | | An input of the new kind 6 holds the arguments a view takes: fields as a message has, no handler, the view's number where a key input names its key, and the view's name with .args after it; one set a view, never a column's (a table's columns and filter read the table's). The view, its columns and filter, and the folds in them read their arguments with FIELD, verified as a handler reads its message (IN_ARGUMENTS: reads, no reply or document), and a fold anywhere may now read the message or arguments of the block it is in: a handler's count(a in accounts where a.bid == pay.bid) verifies. pure_app_arguments gives a view's arguments as a payload of their shape (placed as a message's fields); the view shows nothing until they are given, and a filtered table keeps its rows for the arguments they were made for. A text argument a view shows stays in the caller's payload (FROM_ARGUMENTS), so it outlives the evaluation. A fold reading a number argument still runs a leaf at a time (the same value for every item), and a shaped fold takes arguments only on its key's side, so its pass does not depend on them. Message fields are placed by one function for messages and arguments. view_arguments checks a keyed lookup by argument, a count above an argument for 43 values against a model, a text argument shown back as it is, a table filtered by an argument whose column folds with it (rows kept per arguments), a handler's fold over its message, and five refusals.
* Take a message's rows over HTTPeunhhu2026-09-288-30/+421
| | | | | | | | | | | | | | | | | | | | | | | A rows field's JSON value is an array of objects naming the record's fields (numbers, strings; a field left out takes its default). The connection thread encodes them as the list's stored form, which may be longer than the JSON, grows the request's input by that much (pure_runtime_grow: charged to the payload budget as admission charges an input, 503 payload_budget when it does not fit) and places them after the body, so the owner still applies a message it need not parse. A message whose only variable field is rows takes a body that is not JSON as those rows in their stored form: a bulk insert without JSON. An API's answer (--reply) gives rows the same way. A delivered batch holds no rows. server_rows loads 30,000 stock rows in one JSON request (some 250,000 rows a second, against 6,500 with a request a row) and 20,000 more as bytes, refuses a load with a key already there whole (409 exists), then checks 400 random orders against a model (294 applied, 106 refused with none of their rows applied), texts with escapes and Korean, a walk that removes, and a restart that keeps every state and no rows. The format's document (008) describes walks, rows fields and lists of rows, and OUTER_ROW.
* Walk a list in a handler, and fill a list with a message's rowseunhhu2026-09-285-47/+393
| | | | | | | | | | | | | | | | | | | | | | | | | | | | EACH runs the block that follows in a handler once for every item its list has when it begins, in order, with ROW the item: the block may change states, the list's items (a field, or removing one) and other lists, but may not insert into the list it walks, hold another walk or reply, so a walk runs at most its items times its block. Removing the walked item, or one before it, moves the walk with it, so no item is skipped. A fold in the block reads the walked item as OUTER_ROW, as a fold in a column reads its row. SQL's UPDATE ... WHERE and DELETE ... WHERE are a walk. A message field may now be rows (type 5, naming a list state): records laid out as the list's stored form, filling that list, whose flags hold PURE_LIST_ROWS and whose name is the field's path (order.lines), when the message arrives; rows that are not whole fail it (limit). A list of rows is emptied when every event ends, is never saved, and its changes are not logged for undoing, so a handler walks, folds and counts a message's rows as any list, and may use one as a scratch table within its event. An output has no rows. pure_list_clear lets a list's items go. walks checks a shop against a model over 3,000 random events: orders whose rows each take stock and are logged, all or nothing (997 applied, 1,322 failed and undone, by an item not stocked or not enough of it), and a clean that walks the stock removing items (83 while walking, 32 of them before the walked item). Nine refusals are checked, and 20,000 damaged walk modules run safely with no rows left after any event; all under ASan and UBSan too.
* Record queries on the bank beside SQLite (091)eunhhu2026-09-284-3/+122
| | | | | | | | | | | | 091 measures four queries on pgbench's tables of 100,000 accounts, the same rows and answers on Pure (registered queries, HTTP and JSON included) and SQLite 3.46.1, through each step: items field by field in leaves, folds and filters a leaf at a time, and one pass answering a fold in a column for every row. Per-branch totals went from 398 ms to 4.9 ms (SQLite 45, 14.3 with a DBA's indexes), a three-table join from 3.3 to 0.48 ms (1.47), and each rich account's rank from 9.2 s to 6.8 ms (2.84 s, 7.6 ms with an index); counting accounts in credit takes 1.05 ms where an index lets SQLite count 0.16 ms. Pure declares no index.
* benchmarks, docs: add bank queries benchmark and outside feedbackeunhhu2026-09-283-0/+269
|
* Answer a fold in a column from one pass over its listeunhhu2026-09-285-17/+474
| | | | | | | | | | | | | | | | | | | | | | | | | | | | A fold in a table's column runs once per row, over its whole list each time: per branch, the sum of its accounts read all 100,000 accounts for every branch, and per rich account, how many are richer read them all for each of 404 accounts. At load, a planned fold in a column whose test compares a field of its item with a key the row gives, the rest of it reading only its list, its item and constants, gets a shape: a group (the field equals the key) or a rank (a count of items whose field orders against the key, and nothing else). The app then keeps one pass over the list per fold and list version: a group pass runs the block a leaf at a time with the comparison taken as true and the key left out, folding each item into its field's group; a rank pass radix-sorts the field's values. A row runs its key alone and looks it up. The pass is made at the second row that asks, so a fold asked once scans once. Answers are the fold's own: items join their group in order, so a sum adds as it would; a key that faults leaves that row to run the fold as written, and an item that would fault in the pass leaves every row to. lane_folds now also makes shaped folds (185 groups, 127 ranks, 98 passes made) and checks them against the same module run item by item. On the bank of 100,000 accounts: per-branch totals 4.8 ms (16 ms before, SQLite 46 ms, 14 ms with the index a DBA would add on branch and balance) and each rich account's rank 6.7 ms (268 ms before, SQLite 2.84 s, 7.6 ms with an index on balance), HTTP and JSON included on Pure's side. Pure is given no index.
* Run folds and filters a leaf of their list at a timeeunhhu2026-09-285-48/+685
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A fold ran its block once per item through the interpreter, a call and a lookup of the item each: counting 100,000 accounts took 4.7 ms and a per-account rank over them 5.3 s. When every instruction of a fold's block (or a table's filter) has a lane form, as numbers and indexes with no branch and no text do, the module marks it at load, and the block then runs for a leaf's items at once: a value is one per item (a leaf's column read in place, or computed into a row per stack depth), the items' positions, or one value for all. Lookups by a value for all items are made once per fold, not per leaf. Leaves are walked with an iterator that keeps its path (pure_list_next) and fetches the next leaf ahead. The lanes only take what the interpreter would compute. Where an item would fault (division by zero, a non-finite result, a key not found, an index past the end, a clamp the wrong way round, a fractional move), the whole leaf runs again item by item, so a fold faults, and a filter leaves the item out, exactly as before; items join the fold in order, so a sum adds as it did. lane_folds checks it: 300 rounds of random well-typed folds in views, in a table's columns reading their row and in a handler storing them, with a random filter, over random lists, answer bit for bit as the same module run item by item, 920 faults included by kind and instruction. list_trees now walks runs too. On the bank of 100,000 accounts (benchmarks/bank/queries.py), against the same SQL on SQLite 3.46.1: per-branch totals 16 ms (SQLite 46 ms), accounts in credit 1.1 ms (6.2 ms), large transfers with their accounts and tellers 0.77 ms (2.1 ms), each rich account's rank 268 ms (2.85 s), HTTP and JSON included on Pure's side.
generated by cgit v1.2.3 (git 2.46.0) at 2026-10-20 16:10:44 +0000