aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* Guide a list's inner nodes: items before each child and its first key, ↵eunhhu2026-09-284-44/+197
| | | | searched without branches
* Save a list as its items and a journal of its changes sinceeunhhu2026-09-288-186/+358
|
* Run TPC-C on Pure and on SQLite, and compare the states they leaveeunhhu2026-09-285-0/+1316
|
* Allow 16 records and 128 record fields a moduleeunhhu2026-09-282-2/+2
|
* Slice a keyed list: fold and walk over a range of keyseunhhu2026-09-284-29/+87
| | | | | | | | | | | | | | | | A fold whose kind has PURE_FOLD_SLICED (128), and a walk whose argument has PURE_EACH_SLICED (bit 0), first take two numbers, a low and a high key, and run for the items of their list (keyed by a number) whose keys are from low up to before high, found by key: the range is the program's meaning, not an optimization, so their blocks may fault (items out of the slice never run). A fold a leaf at a time now leaves out the lanes of a leaf outside the items it runs for, and a leaf that would fault runs item by item over those items alone. Slices are neither shaped nor given automatic ranges. ranges now checks a count over a slice whose test divides (a division by zero failing it only when inside the slice) against a model, the same on the item-by-item path, and a walk over a slice.
* Read only the keys a test allows, and count distinct valueseunhhu2026-09-284-35/+410
| | | | | | | | | | | | | | | | | | | | | | | | | | | A filter, a fold or a walk's where (a test at the start of its block and a jump to its end when it fails) that ANDs comparisons of a keyed list's key with values reading no item gets a range at load (PureRange: a lower bound GT or GE, an upper LT or LE, EQ both). When it runs, the bounds are found once and looked up by key, and only the leaves between the two positions are read: O(log n + k) where it read every item. A filter leaves the items out of range out anyway, so it always may; a fold or a walk only when the rest of its test and its value cannot fault (no division, sum or product that may overflow, key or item that may be missing), since the items it skips would have run them; a walk's bounds must also read no state, local or list its block writes, as the where reads them anew for every item. A bound that faults leaves the test to run for every item, as written. A fold of kind 5 (distinct) counts the different values its included items give (-0 as 0), with a set per fold; it runs a leaf at a time and is never shaped. max, min and first alone fault over no item. ranges checks, against the same module reading every item, over 60 lists of up to 400 keyed items and 3,600 views: a count between two bounds, a sum between a lower and an upper bound with another condition, first equal to a key, min with the bound on the right, a count whose test may fault (no range), a filtered table in bounds, 359 walks changing and removing the items in bounds, and a walk whose body moves its bound (no range). lane_folds now makes distinct folds too.
* Give an event localseunhhu2026-09-285-20/+156
| | | | | | | | | | | | | | | | | LOCAL_LOAD and LOCAL_STORE name one of an event's 64 slots, what it holds (a number, or an index of a list) and that list: slots start at 0 when an event begins, are shared by its handlers, the walks in them and the folds in those, and are never saved nor in the states' namespace. A handler uses a slot for one thing, so its type is verified as a state's is; views have no locals (IN_LOCALS: a handler's block, a walk in it, a fold in either; stores only in a handler and its walks). A fold reading a local still runs a leaf at a time (one value for every item) and a shaped fold takes one only on its key's side. The signature table's where widens to 16 bits. locals checks a handler that sums items into a local in a walk, counts in a fold the items below it, keeps an index in a local and reads it back, over 80 events against a model, and five refusals. server_rows' shop now takes stock through an index local and counts in one.
* Take a view's arguments from the request's queryeunhhu2026-09-283-14/+230
| | | | | | | | | | | | | | | | | | | | GET /views/name?argument=value gives a view that takes arguments its arguments: the query as it came is kept on the request's work slot (it outlives the connection, which may go before the view is shown), and when the view is shown, on the connection thread from a snapshot or by the owner, its pairs (values %XX decoded) are laid out as a payload of the arguments' shape and given before the view is evaluated. from and count still page a table. An argument missing, unknown, given twice, not of its type or badly escaped answers 400 invalid. A registered query takes its view's arguments the same way (GET /queries/name?k=4). An arguments input is not an input to post to. server_arguments opens 200 accounts, then asks 83 times against a model: balances by key (404 no_value for a key not there), counts above an argument, texts with &, =, % and non-ASCII shown back, a table filtered by two arguments whose column folds with one, paged, counts by an owner's name, a registered query with an argument, and six refusals. The format's document (008) describes arguments.
* Let a view take argumentseunhhu2026-09-283-55/+289
| | | | | | | | | | | | | | | | | | | | | | | | | | | An input of the new kind 6 holds the arguments a view takes: fields as a message has, no handler, the view's number where a key input names its key, and the view's name with .args after it; one set a view, never a column's (a table's columns and filter read the table's). The view, its columns and filter, and the folds in them read their arguments with FIELD, verified as a handler reads its message (IN_ARGUMENTS: reads, no reply or document), and a fold anywhere may now read the message or arguments of the block it is in: a handler's count(a in accounts where a.bid == pay.bid) verifies. pure_app_arguments gives a view's arguments as a payload of their shape (placed as a message's fields); the view shows nothing until they are given, and a filtered table keeps its rows for the arguments they were made for. A text argument a view shows stays in the caller's payload (FROM_ARGUMENTS), so it outlives the evaluation. A fold reading a number argument still runs a leaf at a time (the same value for every item), and a shaped fold takes arguments only on its key's side, so its pass does not depend on them. Message fields are placed by one function for messages and arguments. view_arguments checks a keyed lookup by argument, a count above an argument for 43 values against a model, a text argument shown back as it is, a table filtered by an argument whose column folds with it (rows kept per arguments), a handler's fold over its message, and five refusals.
* Take a message's rows over HTTPeunhhu2026-09-288-30/+421
| | | | | | | | | | | | | | | | | | | | | | | A rows field's JSON value is an array of objects naming the record's fields (numbers, strings; a field left out takes its default). The connection thread encodes them as the list's stored form, which may be longer than the JSON, grows the request's input by that much (pure_runtime_grow: charged to the payload budget as admission charges an input, 503 payload_budget when it does not fit) and places them after the body, so the owner still applies a message it need not parse. A message whose only variable field is rows takes a body that is not JSON as those rows in their stored form: a bulk insert without JSON. An API's answer (--reply) gives rows the same way. A delivered batch holds no rows. server_rows loads 30,000 stock rows in one JSON request (some 250,000 rows a second, against 6,500 with a request a row) and 20,000 more as bytes, refuses a load with a key already there whole (409 exists), then checks 400 random orders against a model (294 applied, 106 refused with none of their rows applied), texts with escapes and Korean, a walk that removes, and a restart that keeps every state and no rows. The format's document (008) describes walks, rows fields and lists of rows, and OUTER_ROW.
* Walk a list in a handler, and fill a list with a message's rowseunhhu2026-09-285-47/+393
| | | | | | | | | | | | | | | | | | | | | | | | | | | | EACH runs the block that follows in a handler once for every item its list has when it begins, in order, with ROW the item: the block may change states, the list's items (a field, or removing one) and other lists, but may not insert into the list it walks, hold another walk or reply, so a walk runs at most its items times its block. Removing the walked item, or one before it, moves the walk with it, so no item is skipped. A fold in the block reads the walked item as OUTER_ROW, as a fold in a column reads its row. SQL's UPDATE ... WHERE and DELETE ... WHERE are a walk. A message field may now be rows (type 5, naming a list state): records laid out as the list's stored form, filling that list, whose flags hold PURE_LIST_ROWS and whose name is the field's path (order.lines), when the message arrives; rows that are not whole fail it (limit). A list of rows is emptied when every event ends, is never saved, and its changes are not logged for undoing, so a handler walks, folds and counts a message's rows as any list, and may use one as a scratch table within its event. An output has no rows. pure_list_clear lets a list's items go. walks checks a shop against a model over 3,000 random events: orders whose rows each take stock and are logged, all or nothing (997 applied, 1,322 failed and undone, by an item not stocked or not enough of it), and a clean that walks the stock removing items (83 while walking, 32 of them before the walked item). Nine refusals are checked, and 20,000 damaged walk modules run safely with no rows left after any event; all under ASan and UBSan too.
* Record queries on the bank beside SQLite (091)eunhhu2026-09-284-3/+122
| | | | | | | | | | | | 091 measures four queries on pgbench's tables of 100,000 accounts, the same rows and answers on Pure (registered queries, HTTP and JSON included) and SQLite 3.46.1, through each step: items field by field in leaves, folds and filters a leaf at a time, and one pass answering a fold in a column for every row. Per-branch totals went from 398 ms to 4.9 ms (SQLite 45, 14.3 with a DBA's indexes), a three-table join from 3.3 to 0.48 ms (1.47), and each rich account's rank from 9.2 s to 6.8 ms (2.84 s, 7.6 ms with an index); counting accounts in credit takes 1.05 ms where an index lets SQLite count 0.16 ms. Pure declares no index.
* benchmarks, docs: add bank queries benchmark and outside feedbackeunhhu2026-09-283-0/+269
|
* Answer a fold in a column from one pass over its listeunhhu2026-09-285-17/+474
| | | | | | | | | | | | | | | | | | | | | | | | | | | | A fold in a table's column runs once per row, over its whole list each time: per branch, the sum of its accounts read all 100,000 accounts for every branch, and per rich account, how many are richer read them all for each of 404 accounts. At load, a planned fold in a column whose test compares a field of its item with a key the row gives, the rest of it reading only its list, its item and constants, gets a shape: a group (the field equals the key) or a rank (a count of items whose field orders against the key, and nothing else). The app then keeps one pass over the list per fold and list version: a group pass runs the block a leaf at a time with the comparison taken as true and the key left out, folding each item into its field's group; a rank pass radix-sorts the field's values. A row runs its key alone and looks it up. The pass is made at the second row that asks, so a fold asked once scans once. Answers are the fold's own: items join their group in order, so a sum adds as it would; a key that faults leaves that row to run the fold as written, and an item that would fault in the pass leaves every row to. lane_folds now also makes shaped folds (185 groups, 127 ranks, 98 passes made) and checks them against the same module run item by item. On the bank of 100,000 accounts: per-branch totals 4.8 ms (16 ms before, SQLite 46 ms, 14 ms with the index a DBA would add on branch and balance) and each rich account's rank 6.7 ms (268 ms before, SQLite 2.84 s, 7.6 ms with an index on balance), HTTP and JSON included on Pure's side. Pure is given no index.
* Run folds and filters a leaf of their list at a timeeunhhu2026-09-285-48/+685
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A fold ran its block once per item through the interpreter, a call and a lookup of the item each: counting 100,000 accounts took 4.7 ms and a per-account rank over them 5.3 s. When every instruction of a fold's block (or a table's filter) has a lane form, as numbers and indexes with no branch and no text do, the module marks it at load, and the block then runs for a leaf's items at once: a value is one per item (a leaf's column read in place, or computed into a row per stack depth), the items' positions, or one value for all. Lookups by a value for all items are made once per fold, not per leaf. Leaves are walked with an iterator that keeps its path (pure_list_next) and fetches the next leaf ahead. The lanes only take what the interpreter would compute. Where an item would fault (division by zero, a non-finite result, a key not found, an index past the end, a clamp the wrong way round, a fractional move), the whole leaf runs again item by item, so a fold faults, and a filter leaves the item out, exactly as before; items join the fold in order, so a sum adds as it did. lane_folds checks it: 300 rounds of random well-typed folds in views, in a table's columns reading their row and in a handler storing them, with a random filter, over random lists, answer bit for bit as the same module run item by item, 920 faults included by kind and instruction. list_trees now walks runs too. On the bank of 100,000 accounts (benchmarks/bank/queries.py), against the same SQL on SQLite 3.46.1: per-branch totals 16 ms (SQLite 46 ms), accounts in credit 1.1 ms (6.2 ms), large transfers with their accounts and tellers 0.77 ms (2.1 ms), each rich account's rank 268 ms (2.85 s), HTTP and JSON included on Pure's side.
* Keep a list's items field by field in its leaveseunhhu2026-09-282-239/+369
| | | | | | | | | | | | | | | | | | | | | | | An item was its own allocation (a share stamp and a cell per field) that a leaf pointed to, so reading a field of items one after another chased a pointer to wherever each was made: a count over 100,000 accounts built by HTTP took 14.5 ms, against 4.6 ms after a restart laid them out in order. A leaf now holds its items' cells field by field, numbers as binary64 arrays and texts as spans, and pure_list_run gives a leaf's columns to read item after item. Copy on write moves to the leaf: a change copies the path including its leaf, whose texts stay shared; texts carry the share stamp items had, so one a snapshot may read is retired, not freed. An event's log keeps a removed item's cells until the event is kept or undone. A node split at the end of the list leaves the full node full, so lists that grow at their end (histories, keyed lists filled in key order) fill their leaves. An account of three numbers and a text now takes 41 bytes in a leaf, not some 130 across its allocation and the leaf's pointer. The same count now takes 4.7 ms however the list was built, below SQLite's 6.1 ms on the same rows. list_trees passes unchanged, and the app model's tests under ASan and UBSan.
* Register queries and show them from snapshotseunhhu2026-09-282-17/+270
| | | | | | | | | | | | | | | | PUT /queries/NAME takes a module compiled with aup query: the node loads and verifies it, checks that its states are the app's (names, types, records) and that it has a view NAME, and keeps it (64 at most; 409 for another app's module, 400 for bytes that are no module, 404 without the view). GET /queries/NAME?from=&count= shows that view from the last save's snapshot with a reader of the query's module, as a declared view is shown, so a query runs beside writes and sees one save whole. A query registered again answers anew. server_queries registers a per-branch total (a correlated fold), a join of history rows with their accounts and tellers, each rich account's rank and a filtered count, checks them against a model of the same transfers, asks while transfers go on, pages, and checks the refusals.
* Let a fold that runs in a column read the column's roweunhhu2026-09-282-17/+32
| | | | | | | | | | OUTER_ROW pushes the row of the column a fold's block runs for, as an index of the column's list; ROW inside the block stays the fold's own item. The verifier accepts it only in a fold's block inside a column, naming that column's list (a fold in a handler, a plain view or a filter has no outer row). So a column can hold a correlated fold: per branch, the sum of its accounts; per account, how many are richer. aup compiles the column's row inside a fold to it.
* Run pgbench's transaction on a bank of 100,000 accounts, beside SQLite (090)eunhhu2026-09-279-9/+511
| | | | | | | | | | | | | | | | | benchmarks/bank/bank.aup keeps accounts, tellers and branches as keyed lists and a history list; a transfer changes a row of each and appends a history row in one event, as pgbench's TPC-B-like transaction does in SQL. run.py makes a scale's rows and drives transfers through the load generator's new transfer kind; sqlite-bank.c runs the same tables and SQL on SQLite 3.46.1, a connection a thread, WAL. With lists as trees, transfers run at 30.5k a second on 1,000 accounts and 24.5k on 100,000 (RAM store, one connection thread, HTTP and JSON included), where they ran at 3.2k on 30,000 before. Durable on the SD card, Pure makes 1,339 a second and SQLite with synchronous=FULL 84 to 153. SQLite in process with no HTTP makes about 40k in RAM: recorded as it is, with the write amplification of 100,000-row tables on the card and ad hoc queries as what comes next.
* Let a runtime-owned state grow past the document limiteunhhu2026-09-272-2/+13
| | | | | | | | | A list's stored form is one value in the tree store, so the limit on a document (1.38 MiB, what a host carries whole as an HTTP body) capped a table at about 51,000 rows of 28 bytes. A value the runtime owns, a state (a key starting with '.') or a draft without a key, may now grow to 1 GiB; documents keep their limit. The store's format is unchanged: its trees already reach far past either.
* Keep a list's items in a persistent treeeunhhu2026-09-273-109/+673
| | | | | | | | | | | | | | | | | | | | | | | A list was one array of item pointers: an item's place in the stored form was found by adding up the sizes of every item before it, so each change and each appended row cost O(n); a snapshot copied a changed list's whole array at each save; and a list held at most 65,536 items. A bank of 30,000 accounts ran transfers at 299 us of CPU each, and set up in time growing with the square of its rows. Items now live in a B+tree of fanout 32 whose nodes know the items below them, their stored bytes and the first of them: positions, keys (keyed lists) and stored offsets are found in O(log n). Nodes carry the list's share count when made, as items do: one a snapshot may read never changes, so a change copies the path to it and retires the nodes it replaced, and a share is a root and a count. Reads in order go through a finger, each thread's last leaf. A list may hold 2^24 items. list_trees checks it against a model array: 6,000 events of inserts, removals and field changes, 1,195 of them undone, up to 8,983 items (three levels), the stored form equal byte for byte, four shares kept whole while the list changes on, and 13,959 keyed items found by key as a binary search finds them. The app model's tests pass under ASan and UBSan too.
* Record the messenger between two nodes and its latency (089)eunhhu2026-09-275-4/+215
| | | | | | | | | | | | | | The roadmap's E2EE messenger made of the runtime's general parts, with no messenger code in it: sealed, authenticated sessions between nodes (086, 088), Vega on each node's display as a surface (design 010), the app in Vega's examples. Vega's check runs two nodes through a recording relay: a line typed on one display is drawn on the other, frames match the vega command's, and the relay sees no text. A line takes 0.79 ms at p50 and 1.1 ms at p99 from one node's program to the other node's views with stores in RAM, both commits, the HTTP delivery and the seal included (sealing adds nothing measurable); on the Pi's SD card 3.5 ms at p50, the long tail being the card's syncs.
* Run a surface for people inside the node (design 010)eunhhu2026-09-278-14/+449
| | | | | | | | | | | | | | | | | | | | | | | pure-serve --surface LIBRARY [-- ARGUMENTS] maps a library and runs its pure_surface_run in a thread of its own (serve/surface.h). The surface reads the app as of the last save where it is, a published snapshot in a reader app, as a connection thread reads views; an eventfd tells it of each new snapshot; and its events (a key pressed, text typed) are requests without a host in the owner's queue, applied and saved with the round, their results handed back to the surface's waiter. So one node serves programs over HTTP and people on its display with one owner of its states, and nothing between the states and the screen. pure-serve exports its pure_* functions for the surface to call, one copy of the app model in the process; the table carries sizeof(PureApp) and sizeof(PureModule) so a surface built for another layout refuses. The check builds a probe surface that types, presses and reads: its events show in HTTP views, an HTTP input shows in it, wrong inputs are refused, it stops with the server, and a module without states cannot have one. Without a surface the request path measured the same (small GETs 140.9k vs 139.9k, app totals 126.3k vs 126.3k over eight rounds); the exported symbols cost 11 KB of the file and 24 KiB idle.
* Seal in one session a wire, authenticated by the sender's key (088)eunhhu2026-09-2713-155/+502
| | | | | | | | | | | | | | | | | | | | | | A sealed request cost three X25519 operations (about 0.5 ms), so sealed sends drained at half the plain rate. A wire now keeps one HPKE context, encapsulated at its first seal, and seals each request as the context's next number (a failed seal or a restart starts a new session); the header carries the session's encapsulated key and the number, so a receiver opens a session once per connection thread (16 remembered, the least used replaced) and each seal by its number, also after it restarts, unasked. A sender with a key of its own seals in HPKE auth mode and names that key in the header; a receiver started with --sender NAME=KEY takes deliveries only from the senders it names, each sealed with its key (403 sender otherwise). The body is now the ciphertext and its tag. Sealed sends, sender authenticated, drain at 461-464k a second against 462-470k plain and 207-234k sealed a request at a time (Raspberry Pi 5). The check adds: a wire's requests share one session numbered from 0; a changed number or sender key does not open; a receiver restarted naming its sender opens the same session again; another node claiming that sender's name with its own key is refused and applies nothing.
* Record the small reads and idle memory work, next to axum and drogon (087)eunhhu2026-09-276-3/+270
| | | | | | | | | | | | What the app model, snapshot reads and delivery had cost small reads (126.7k -> 116.1k GETs a second) and idle memory (2.0 -> 4.5 MiB), how the sampler found it, and the four steps and the crypto split that took it back: 139.9k and 2.3 MiB. Measured again today beside axum 0.8.9 and drogon 1.9.0 on the same Pi: a one-thread pure-serve answers small GETs at 139k/s (2.2x axum and drogon a CPU second), 1 MiB GETs at 2,362/s (axum 1,304, drogon 754), durable app writes at 21.3k/s (both about 200), with the least idle memory of the three.
* Map OpenSSL only in a server that uses iteunhhu2026-09-2711-112/+276
| | | | | | | | | | | | | | | | | | Linking OpenSSL cost every pure-serve 2.3 MiB of resident memory, 0.66 MiB of it the process's own (relocated data), though most servers never reach an https receiver or seal a send. The code that uses OpenSSL (TLS to https receivers, sealing) is now built as pure-crypto.so beside pure-serve and exports one table; pure-serve maps it the first time a wire to an https URL opens or a key is read or a seal wired (serve/crypto.c), and a missing library is reported at startup. What needs no OpenSSL, a key's hex and the suite this CPU seals with, stays in pure-serve. An idle server: RSS 4,496 -> 2,320 kB, PSS 1,639 -> 881 kB, private dirty 1,296 -> 864 kB. A static OpenSSL was measured too and is worse (RSS 3.8 MiB, but 2.3 MiB private: all of its tables relocated). The checks, https webhooks and sealed sends included, pass unchanged.
* Stop reading a connection at a short readeunhhu2026-09-271-3/+17
| | | | | | | | | | | | | | | | | | After answering, a connection went back to recv for its next request and got EAGAIN nearly every time: a failed syscall a request. A recv that returns less than it asked for has emptied the socket, and the edge-triggered epoll reports the next bytes when they come (epoll(7); as nginx does), so the connection now waits for that; a peer that hung up is read to its end as before. Small keep-alive GETs, 64 connections, five rounds in rotation (Raspberry Pi 5): 131.9k/s before this step, 139.9k after. With the three steps before it (a burst's keys in one look, the clock once a round, the head without printf): 116.1k -> 139.9k/s, p99 672 -> 512 us, above the 127.0k of 2026-09-26's build that the app model had since slowed. App reads: the total 111.5k -> 125.4k/s, a 20-row page 86.2k -> 94.3k; 1 MiB reads and durable posts unchanged.
* Write an answer's head without printfeunhhu2026-09-271-9/+45
| | | | | | | | | Three snprintf calls a response (the head, its ETag and a delivery's batch size) spent a sixth of a small read's user time in printf's format scanning. The head is now written part by part, the same bytes as before, clipped at the end of its buffer where snprintf's would-be length could have sent bytes past it.
* Read the clock once a roundeunhhu2026-09-271-8/+11
| | | | | | | | | | A connection's deadline was counted from a fresh clock_gettime on every read and write, three times a request, and every event took the wall clock again: 5.7% of a small read's user time. A connection thread now reads the monotonic clock once after each epoll_wait and deadlines count from that (they are 10 s by default and swept every second), and a round's events share the time it began.
* Find a burst's keys in one lookeunhhu2026-09-274-11/+62
| | | | | | | | | | | | | | | A message for a document scanned the burst's pending keys with strcmp to see whether its key was there, and the host scanned the burst's earlier requests the same way before each one: both quadratic in the burst (64 concurrent reads, about 31 compares each, twice). The app now indexes its pending documents by key (open addressing over an FNV-1a hash, 1 KiB), answers pure_app_pending in one look, and the host asks it; whether the current event's document is pending is a flag instead of a scan by pointer. Bursts split where they did, except that on an owner, which applies a round's reads first, an edit of a key read in the same round now splits the burst at the host rather than the app saving on its own. A unit test checks the index at 200 keys.
* Add a sampling profiler for machines without perfeunhhu2026-09-273-0/+381
| | | | | | | | | | | | benchmarks/sample.c runs a command and samples where its threads spend user time (the task clock every 50 us, one perf_event_open event a CPU, since the kernel maps no ring for an inherited per-task event), then names the samples from the ELF symbol tables of the files the command had mapped: a stripped library from its debug file by build id. On arm64 it also takes the link register, so time in a library's leaf (memcpy, strcmp) is shown again by its caller. The command's user and system time head the report.
* Seal delivered sends to the receiver's key (086)eunhhu2026-09-2712-16/+676
| | | | | | | | | | | | | | | | | | | | --seal OUTPUT=KEY seals each request's body whole to the receiving node's public key with HPKE (RFC 9180, base mode: X25519, HKDF-SHA256, AES-128-GCM where the CPU has AES, else ChaCha20-Poly1305), bound to the request's source, first number and count; X-Pure-Seal names the suite. A receiver's --key is a raw 32-byte file, made 0600 when missing, held in OpenSSL's secure heap; it opens the body where it lies (400 seal when it does not open) and zeroes the plain text when the request's ticket goes. --sealed-only refuses unsealed deliveries (403 unsealed). The check puts a recording relay between two apps: every order applies once, no body shows an item or a field name, a flipped bit is refused, a replayed request applies nothing twice, and a seal to another key does not open. Its hosts choose their own ports (--port 0). Sealed sends drain at 233-235k a second against 469-470k plain (Raspberry Pi 5): three X25519 operations a request, 164 us each. The plain request path measured the same as before.
* Send a delivery's request in one pieceeunhhu2026-09-271-7/+24
| | | | | | | | | | | | A batch went out as a write for its head, then a bracket or comma and the send for each of its sends: two writes a send. Now the head and the whole body are built in one buffer and written once. A receiver stopped while 20,000 sends piled up drained them at 91k sends a second before and at 470k after (three rounds each, Raspberry Pi 5). Writing the head apart from the body measured the same (468k), so the gain is the body's; one buffer is kept as the simpler of the two, and a lone send now leaves in one segment instead of two.
* Bring the checkpoint's open items up to 084 and 085eunhhu2026-09-271-2/+6
|
* Give a compacted file the zeros of its whole cycle (085)eunhhu2026-09-278-10/+274
| | | | | | | | | | | | | | | | Commits grew the tree store's tail 256 KiB at a time, each growth a sync that also committed the ext4 journal for the new size and blocks. A file installed by compaction now carries written zeros up to where the next compaction will come (4 x live + slack, plus a commit's margin), capped at the size of the file it replaces, so a store never takes more space than before and shrinks when its live data does. The cycle's commits then only overwrite blocks the file has. Store commits: 317.5 -> 385.5 a second, ext4 journal commits 109 -> 45 for 4,000 commits; durable app posts +5.3%, their p99 the same within this SD card's noise. The new cycles test keeps sizes still between compactions once the live data is whole, and shows the file shrinking with it.
* Measure what moving saves off the owner would buy (084)eunhhu2026-09-273-2/+146
| | | | | | | | | | | | Nothing a thread can give: with writes saturated the device's one line of syncs is the bottleneck, so zeroing the tail or compacting on another thread only queues that device work behind the commits, and the owner's idle gap is shorter than one journal commit. Built without compaction the store is no faster; without compaction and without tail growth durable posts gain 1.4% and the p99 drops 24% (17.4 to 13.3 ms). New blocks are the tail latency, so the way to it is a store that rewrites its space in place, a ring cleaned forward instead of compacted into new files.
* Measure whether sharding owners by key would add writes (084)eunhhu2026-09-274-0/+427
| | | | | | | | | | | It would not, on this device. One file's syncs or four at once, the SD card completes the same 510-690 commits per second, the latency growing with the number of files. The scoreboard app with two owners, each with a store of its own, took 23% fewer durable posts than one owner (median 16,230 against 20,974 a second) with 2-3x the p99: syncs queue, and each carries half the events. One owner's group commit stays; cores serve reads, and writes grow by making the sync itself cheaper.
* Check pure-serve in Pure, not in aupeunhhu2026-09-2710-9/+1255
| | | | | | | | | | The HTTP host's end-to-end checks (documents against the earlier server, lending, kill -9 durability, JSON, timers, links, webhooks and answers) sat in aup's tests, so a change to the host was verified from the toolchain. They now live in tests/check.py; make check builds their apps with the sibling aup and runs them after the unit tests, which stay beside their parts. Design 009 states the rule for all three repositories.
* Show the new layout costs nothing, and keep code where it waseunhhu2026-09-273-1/+17
| | | | | | | | | | | | The benchmark runs an earlier pure-serve build in turn with the current one (--baseline). Against the build before the move, linking through libpure.a costs nothing: a one-command build with the library's object order has the same size and the same CPU per request. What moved was code placement, so the runtime links last again, as it did; the two orders now differ by less than rounds of one build do (total -0.4%, page +1.2%, mixed -0.1%). The binary is 5,008 B smaller: unused flow.c functions stay in the library, and the linker's A53 erratum padding is gone.
* Lay Pure out as a core with the parts selected around iteunhhu2026-09-2750-79/+222
| | | | | | | | | | | | | | | | | | | | | The app model, the tree store and the HTTP host lived in prototypes/connections among the earlier document runtime, the store was known only by its format's name, and Vega compiled six Pure sources by path. Now, as the README's principle has it: core/ runtime, flow and the app model (module*.c), with what the core asks of storage (state.h, store.h) store/ the tree store (PURETRE3, PUREREC1) serve/ pure-serve: HTTP host, outbound delivery, JSON in place cli/ pure-run, pure-edit-module A top-level make builds libpure.a (app model and store, for hosts to link) and the hosts; make check runs the store's and the app model's tests. The earlier runtime still builds from prototypes/connections against core/ and store/ (check, check-tree and check-core pass). Design 009 records the rule: parts by role behind libraries; git repositories split by coupling, so the store stays here while its contract moves with the core, and Vega stays apart.
* Look for new sends newest first, only as far back as they goeunhhu2026-09-275-13/+45
| | | | | | | | | After each save the owner walked the whole outbox to find the sends not yet queued, so a receiver long down made every save walk the backlog. pure_app_outbox_newest visits saved sends newest first, and the owner stops once every wired output reaches the number it queued last, then queues what it found in order.
* Make an API's answers the app's events (--reply OUTPUT=INPUT)eunhhu2026-09-276-35/+281
| | | | | | | | | | | | | | | | | | | | | The final answer to each send of a wired output (a 2xx, or a 4xx but 408 and 429) becomes a message event of the given input: fields from the answer's JSON object by name, other members skipped; a field the answer lacks from the send's field of that name and type, so the request's own id is there; a u32 http_status field is the status. It is applied with the send's number from the wire's source, before the send is trimmed and in the same save, so an answer that comes again after a restart (the API repeating itself for the Idempotency-Key) applies once. Sends that wait for answers go one per request; answers are saved with the next round, or at once when none waits. A server that writes an answer in pieces holds the second until the first is acknowledged, and a delayed acknowledgement stalled every answer 40 ms: the client now acknowledges at once (TCP_QUICKACK) after each read. Against a payments API that answers 200 or 402, with the sender killed (-9) midway, paid and declined orders matched the API's answers one to one.
* Read JSON messages in message-json.c, for requests and answers alikeeunhhu2026-09-274-222/+319
| | | | | | | | | The in-place JSON reader of pure-serve's request bodies moves into its own file with a small interface (pure_json_message, pure_json_fixed, pure_json_space), so the delivery client can read an API's answers with it. It gains a lenient mode for answers: a member that names no field is skipped, value and all. Requests read as before.
* Deliver to https receivers, verifying their certificates (083)eunhhu2026-09-277-33/+107
| | | | | | | | | | A wire to an https:// URL speaks TLS 1.2 or later through OpenSSL: the receiver's certificate must chain to the system's CAs (or to the file --send-ca names) and name its host (SNI and SSL_set1_host, or the IP address). An untrusted receiver gets nothing: the send stays in the outbox, the wire logs it and tries again every 10 s. pure-serve now links libssl and libcrypto.
* Deliver an app's saved sends over HTTP, in order and in batches (083)eunhhu2026-09-2712-48/+852
| | | | | | | | | | | | | | | | | | | | pure-serve --send OUTPUT=http://host:port/path wires an output: after every save the owner queues the new sends as JSON (fields and groups) and a thread per output posts them in order with X-Pure-Source, X-Pure-Sequence and Idempotency-Key. A 2xx, or a Pure receiver's refusal, delivers; the owner then trims. 408, 429, 5xx and no answer are sent again after a pause growing to 10 s, other answers every 10 s: nothing is dropped. A Pure receiver applies a delivered message with pure_app_deliver (now taking placed fields) and advertises X-Pure-Batch: 256; the sender then posts up to that many numbers in a row as one JSON array (X-Pure-Count), applied in order in one burst with one save. Delivery kept up with orders once batched (391 deliveries/s one at a time; a backlog of 20,000 went at 60k/s). With four clients ordering, killing the receiver and then the sender (-9) left the receiver's count and total equal to what the sender saved, five times and under ASan.
* Send messages as part of events and apply delivered ones once (078)eunhhu2026-09-278-18/+742
| | | | | | | | | | | | | | | | | | | | | | | An output (input kind 5) is a message the app sends; SEND (72) takes an output's field values in order in a handler, lays them out as a message's payload and keeps it, numbered per output from 1, with the event's changes: a failed event sent nothing, and a failed save takes back every send since the last save with its numbers. A host sees only saved sends (pure_app_outbox) and trims what its receiver has saved (pure_app_trim, gone with the next save). pure_app_deliver applies a message from a named source once: a number heard before succeeds without applying, a refused message counts as heard, a failed save forgets what it heard. Sends, numbers and heard numbers are the app's links state, stored whole in one runtime document (PLK1) with the states, in the same commit. Outputs follow their names across module changes; a module that drops or retypes an output with sends left is refused, and a dropped output's number is kept for when it returns. Killing the process at every commit stage leaves an event and its send, or a delivery and its effect, together or neither. pure-serve answers a post to an output or a timer with 404 no_input.
* Choose the connection threads with --threads autoeunhhu2026-09-272-1/+15
| | | | | | | | One connection thread for each CPU the process may run on (its affinity, or the online CPUs), less the one the owner of an app with states takes: three CPUs give two connection threads and the owner. The default stays one thread.
* Gather after a save until a just-answered client has come backeunhhu2026-09-272-8/+21
| | | | | | | | | | | | | After a round that saved, the owner waited for 100 us without a new request (1 ms at most) before the next round. Requests that queued up during the save made that quiet start at once, so the clients just answered missed the next round; the clients then settled into two halves taking turns, each save carrying half of them. It showed with two connection threads: 34 events a round and 15.8-16.0k posts/s with 64 connections, against about 61 and 22-24k/s with one or three. The quiet now counts only once a request has come after the answers were handed back: two threads carry 61.6 events a round, 22.9-23.9k/s.
* Record the bound of taking tail growth and compaction off commits (081)eunhhu2026-09-271-3/+5
| | | | | | | Measured with a build that grows its tail by 64 MiB once and never compacts: durable posts +5-8% (64 connections), p99 about halved; the slow syncs that remain come from the device.
* Read JSON bodies in place, groups included (082)eunhhu2026-09-279-79/+260
| | | | | | | | | | | | | | | | | | | | | pure-serve no longer copies a JSON body's strings into the message's payload. The ingress is the prefix, a table of each field's start and size, and the body as it came; the connection thread scans the body once, puts numbers into the prefix and records where each string is. Only a string with escapes is unescaped, over itself (it never grows). A group of fields comes as an object inside the body and, for a fixed field, as its path in the query (?to.zip=7). A group where a field belongs, a value where a group does, an array or true/false/null is json_type; an unknown member, inside a group too, json_field. The string scan and UTF-8 checks now pass over eight bytes at a time while none needs a look (pure_utf8_step, so every text check gains). On the Pi, posting {"name":"k","text":"x..."} to a stateless app went from 652-655 to 3,737-3,953 requests/s with 256 KiB of text (server CPU 1.52 to 0.25 ms a request, peak RSS 5.2 to 3.2-3.4 MiB) and from 29.1k to 76.3-76.7k/s with 4 KiB; the load generator gains a json method and benchmarks/http/echo.aup is the app.
generated by cgit v1.2.3 (git 2.46.0) at 2026-10-20 23:50:01 +0000