aboutsummaryrefslogtreecommitdiffstats
path: root/docs/design
Commit message (Collapse)AuthorAgeFilesLines
* Design 022: stored lists as arrays with a bucket index fitted to the keys ↵eunhhu2026-10-051-0/+151
| | | | searched, narrow columns, a JIT with registers and loop guards, read ahead; criteria first
* Measure 021: BFS 190 to 90 ms over HTTP against C's 9.1, short of C; Pure's ↵eunhhu2026-10-051-1/+2
| | | | forms by hand beat C only packed, narrow and read ahead (123)
* Measure an index for a stored list's slices outside the list: three times a ↵eunhhu2026-10-051-3/+8
| | | | tree's seek, still seven times C's (021)
* Say what a search's floor is with stored lists as trees: 373 ns a node to ↵eunhhu2026-10-051-1/+16
| | | | find a slice against C's 21 (021)
* Say how a search gets past C: event lists as arrays, a directory for stored ↵eunhhu2026-10-051-0/+112
| | | | keys, then a JIT, with criteria (021)
* The same answers on the final build and the same pure-runs on the Pi, WSL2 ↵eunhhu2026-10-041-2/+3
| | | | and a Mac (121); 019 points at 020's view workers
* Say 020 is built and measured (121, 122)eunhhu2026-10-041-1/+1
|
* Compute views with a body on view workers, off the connection threads, asked ↵eunhhu2026-10-041-0/+11
| | | | | | | | | | | | | | | | | | | | and pushed (020) - pure-serve --view-workers N (a CPU each; 0: on the connection threads): a GET of a view with a body (or a table with such a column) goes to the workers, which show it from the app's newest snapshot on readers of their own and give the answer back as an owner does (eventfd). The connection thread reads and writes its other connections meanwhile. - A snapshot of another module than the request was resolved with, or none (the app stopped), answers 503 stopped. Past an app's queue=N of them, 503 busy. - A push group of such a view hands one evaluation at a time; a newer snapshot meanwhile is evaluated after it; a group whose subscribers all left stays until its evaluation is back. - check.py server_view_workers: 8 askers of a long view while its app changes module (another view in its place) and then goes: every answer the old module's, the new one's, 503 or 404; streams closed at once leave nothing; --view-workers 0 as before.
* Keep one finger a thread, as before: four cost the bank's queries ↵eunhhu2026-10-041-4/+6
| | | | instructions and a search no time (020)
* Search an event's keyed lists by a key set and lists by fingers, add by key ↵eunhhu2026-10-041-1/+18
| | | | | | | | | | | | | | | | | | | | in one descent, and loop without recursion (020) - An event's list keyed by a number keeps its keys beside its tree: bits while they are small whole numbers, else an open-addressing table; has() reads only that, and inserts, removes, key changes, their undoing and clears keep it. - Each thread keeps fingers on the last four leaves it read (list id and version must match): reads by position and searches by key that land in them do not descend. - pure_list_add finds a key's place and inserts there in one descent; a stored keyed list's appends take it too, journaled as inserts. - Child searches count over the whole guide; item fields know at load whether they are text; loops run on a stack of loop records, not by recursion; module-list.c builds at -O3. - test-module: the key set and fingers answer as a model at every step (bits to table, removes inside runs, undone changes, clears, shares), and a stored keyed list's adds replay from its journal.
* Say where a search's time goes and how to make it and long views fast, with ↵eunhhu2026-10-041-0/+84
| | | | criteria (020)
* Say what pushing views costs and how fresh it keeps them (019, research 120)eunhhu2026-10-041-1/+1
|
* Push a view's value to its subscribers over SSE, once per group and interval ↵eunhhu2026-10-042-1/+10
| | | | (019)
* Fix 019 before building: compare equal freshness, bound pushes per interval, ↵eunhhu2026-10-041-9/+24
| | | | carry streams across module changes
* Say how pure-serve pushes a view's value to its subscribers, and what it ↵eunhhu2026-10-041-0/+91
| | | | must show (019)
* Say what views that compute cost the writes beside them (018 stage 3, ↵eunhhu2026-10-041-1/+2
| | | | research 119)
* Say why stage 3's first isolation criterion could not hold, and what to ↵eunhhu2026-10-041-0/+21
| | | | measure instead (018)
* Let a view compute in its body, on a reader's own lists (018 stage 3)eunhhu2026-10-042-5/+16
|
* Say what computing in one event costs and finds (018 stage 2, research 118)eunhhu2026-10-041-1/+2
|
* Count in loops that nest, break, and keep an event's own lists, every event ↵eunhhu2026-10-042-10/+30
| | | | bounded (018 stage 2)
* Say what a transient state saves a write (018 stage 1, research 117)eunhhu2026-10-041-1/+2
|
* Keep transient states the app's while it runs, never stored (018 stage 1)eunhhu2026-10-042-5/+13
|
* Say how aup chooses what a state outlives, and computes in memory (018)eunhhu2026-10-041-0/+199
|
* Say what an app's tokens cost (017 3b, research 116)eunhhu2026-10-041-1/+9
| | | | | | | | Requests to an app that checks tokens go 2.92% slower than to an open one on 64 keep-alive connections, 2.2% of it the head the token makes longer (an open app sent the same head is as slow); hashing every request on a new connection costs 0.91%. 031 measures the three in turns.
* Say how an app's routes take tokens, and what it must show (017, 3b)eunhhu2026-10-041-1/+30
| | | | | | | | | An app given a tokens file (tokens=FILE, --tokens FILE) takes requests bearing a token whose SHA-256 the file lists: read for its views, queries and documents, write for all, input=NAME for one input alone; 401 without one, 403 past its grant. A delivery another node sealed needs none: its key told who sent it. A connection keeps its last token's grant; a reading swaps the tokens without stopping the app.
* Say that reading the code found stage 3's two races, and how a host changes appseunhhu2026-10-041-2/+2
| | | | | | | | | The use after free in local_done and the owner's lock made after another app's outlet used it were found reading the code, not by the check, which passed with both before it was made to change senders (115, 017). The usage now says changed lines change in place unless their store moved; and 115 names the path no check takes: a module refused while changing in place.
* Say what apps that change on a running host showed (017 stage 3, research 115)eunhhu2026-10-041-6/+24
| | | | | | | | | | Changing an app in place keeps a million items' app answering but for 1.12 ms (a host restart: 150 ms; stopping and starting it again from its store, as first built, 150.4 ms), 171.5 ms when its records change shape (332 ms). Another app's p99 stays 0.97 times as it was while one changes every 0.2 s; a host of one app is as it was (TPC-C +0.7%, instructions at most +0.02%); each module change leaves 19 KB. 030 holds the windows of both builds, the isolation, the pairs, the counts and the memory.
* Say how apps start, stop and change on a running host, and what it must show ↵eunhhu2026-10-041-1/+76
| | | | | | | | | | | | (017, stage 3) The --apps file is the host's shape: on SIGHUP the host reads it again and starts the apps of new lines, stops those of lines gone, starts again those whose line changed (states carried by name from the store) and those stopped or uncertain. A request carries the module it was resolved with, and is not applied under another (503 stopped); link streams and local outlets look their input up again. Route authentication (3b) is designed apart.
* Say what one sending loop showed (017 2b, research 113)eunhhu2026-10-011-6/+22
| | | | | | | The loop alone slowed four links to half: woken by every record queued, it chased the owner a record at a time. Pushing once a turn made links 1.6 times as fast, with threads or the loop alike; the loop's own cost is within the noise. Twenty outputs now add two threads, not 21.
* Say how sends will share one loop, and what it must show (017, 2b)eunhhu2026-10-011-3/+18
| | | | | | | | | | | | | | | | Each link and each HTTP wire sends on a thread of its own: a host with twenty outputs runs twenty-one threads more than one without. 2b moves them into one sending loop: an epoll over the links' UDP sockets and the wires' TCP sockets with one wake, waiting until the soonest link timer or wire pause. Links (2b-1) do in the loop what their threads did each round; wires (2b-2) become a non-blocking HTTP/1.1 client state machine, with non-blocking steps added to the crypto table for TLS. A name looked up again still waits where it is, and says so. Before building: every link, wire, answer and seal check green; a host of ten links and ten wires two threads above one without; and link and wire throughput, alternated before and after, no more than 5% lower at the median (four links together: 10%).
* Say that apps of a host send to each other and to other nodes (017)eunhhu2026-10-011-2/+13
| | | | | | | | Stage 2 is built and measured: what was learned (the receiving side needed nothing new, an output goes one way only) and the numbers. The checkpoint adds sending to the many-apps row, and keeps 2b (one loop for the send threads) and stage 3 (apps added and replaced while the host runs) among what is left.
* docs: link aup and vega via clean web paths instead of raw server pathseunhhu2026-10-011-1/+1
|
* Say how apps will send to each other and to other nodes, and what it must ↵eunhhu2026-10-011-2/+41
| | | | | | | | | | | | | | | | | | | | show (017) Stage 2: each app of --apps takes its wiring (send=, seal=, reply=), its sends' source being APP.OUTPUT; one --link-port takes every app's inputs, a stream's input name APP/INPUT (012 unchanged); and send= OUTPUT=APP/INPUT hands an app's saved sends to another app of the same host with no socket, seal or packet: an outlet with the window links have, a batch of records as a link delivers them put in the receiving app's queue, applied after its last number heard, and the number heard after its save trimming the sender's outbox. Merging the send threads into one loop is split off as 2b. Before building: it goes in if the inner outlet stores at least as many sends a second as a loopback pure:// link between the same two apps, at half the host's CPU a stored send or less; checked by a host sending to an app of its own and to one of another host, exactly once and in order across a kill -9, and held back by a full receiver.
* Say that one host runs many apps, and what its first stage found (017)eunhhu2026-10-011-10/+41
| | | | | | | | | | 017's first stage is built and measured: the shape now says what was built where it differs from the design (work counted as written, nothing counted without a budget, every app of --apps run by workers, a busy app kept on its worker, a home thread for each app's connections), and what was learned on the way. The checkpoint lists many apps on one host among what Pure shows, and stages 2 and 3 among what is left.
* Say how one host will run many apps, and what it must show (017)eunhhu2026-10-011-0/+128
| | | | | | | | | | | | | | | | | | | | | | | pure-serve runs one app a process: a module, a store, its own threads, port and key, scheduled by Linux. Within one process nothing bounds an event's run (it grows with its lists), the requests waiting for the owner, or one store's sync, which holds every other store's commit since the tree store has one writer's turn per process. 017 makes an app an execution domain (007): a host of many apps (--apps FILE, routes /APP/...) runs them by turns on a fixed number of worker threads, none per app; budgets per app in deterministic units (memory in stored bytes as before, work in instructions counted per block and per item visited, queue in requests: 422 work, 503 busy); writer turns per store; and an app that cannot start or whose save became uncertain stops alone. Stage 1 is all that; links, sends between apps and an app's lifetime come after. Before building: it goes in if an app B's p99 under app A's flood is within 1.5 times what two processes give, an app costs under a quarter of an idle process's own memory with no thread of its own, the pair's throughput is within 10% of two processes', and a one-app host keeps its speed (3% in time, 1% in instructions).
* Say that i64 is in, and what it found (016)eunhhu2026-09-301-1/+26
| | | | | | | | | | | 016 is built: bool and i64. The second stage's findings: an integer literal takes the other side's type, found by compiling it on trial (linear once the wanted type is passed down); a literal with no context stays a number, so no program changes meaning and modules without i64 are the same bytes; one rule carries values between numbers, bools and i64s; and the number paths passed the timing criterion, with what i64 had added to them counted and halved (110). The checkpoint lists bool and i64 in the app model and drops i64 from what is left.
* Pin an i64's meaning on every path (015)eunhhu2026-09-301-5/+27
| | | | | | | | | server_meaning gives every value an i64 field, 13 values from 0 and -0 to both ends of the range and just inside them, in over JSON, rows and a query, over the link (where B takes id, a u32, as an i64) and HTTP, and out at all seven exits with every digit. 32 more refusals, each with its reason on JSON, a query and rows; an i64 view's text at both ends and an i64 argument that counts exactly and refuses the rest.
* Give i64 its own type in modules, lists, stores and the host (016 stage 2)eunhhu2026-09-301-7/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | An i64 is an int64_t in a value's 8 bytes, its own stack type: CONST and NUM_TEXT take argument 2 for one, eleven opcodes are appended (ADD, SUB and MUL fail as overflow past its range, DIV and MOD floor and fail as divide by zero, NEG, COMPARE with a relation, MIN, MAX, and I64_FROM and NUM_FROM, which convert only where the value fits and else fail as limit), and moves (LOAD, STORE, locals, items, FIELD, SEND) copy it as they copy a number. Type codes: state 6, record field 3, message field 7 (8 bytes), view 4, local 3. v6 stays: all of it is added. - Folds with PURE_FOLD_INTEGER sum (overflow fails), keep the most or least of, or count the different i64s; distinct's set keeps an i64's -1 apart from its free mark. Blocks with an i64 run item by item. - A list's kinds are its stored header's (number, text, bytes, bool, integer): integer columns are int64_t, ordered as integers as keys (found without guide keys for now), stored in 8 bytes and logged as zigzag varints of any size. Records name their fields' type in one array (record_field_type) in place of text and bool flags. - A number, a bool and an i64 carry over each other by one rule, pure_cell_as, wherever one becomes another: a module replaced, a stored value or item read by another module, a waiting send recast. A u32 field delivers to an i64 one. - The host reads i64 fields from JSON, rows and queries as integer digits in range and nothing else, and shows every digit in views, tables, deliveries and pure-run. Faults are named by one function, pure_fault_name, in place of three tables; overflow is new. test-module: 28 arithmetic cases at the range's ends, relations, conversions both ways, 14 refusals, a keyed i64 list app (order, a sum that overflows, a failed add undone, messages and sends of 8 bytes, every digit as text), recasts, replacement between numbers, bools and i64s, and three stored runs against memory with restarts (46 images written).
* Say how i64 will be built, and what it must not slow down (016)eunhhu2026-09-301-13/+58
| | | | | | | | | | | | | | | | | Before building i64: a value slot holds a number or an i64 in the same 8 bytes (the stack, states, list cells, locals, constants), moves copy the bits and only computing opcodes look at the type; eleven opcodes are appended (arithmetic that fails as overflow, floor division and modulo, compare, min, max, the two conversions), CONST and NUM_TEXT take argument 2, and type codes are added, so v6 stays. i64-keyed lists compare as integers without guide keys for now; folds over i64 run item by item. JSON and queries take plain integer digits in range; views, tables and deliveries show every digit. u32 widens to i64; i64, numbers and bools do not deliver to each other; replacing converts when the value fits. It is in when every check is green and the number paths keep their speed: bank queries and TPC-C (RAM) no more than 3% slower at the median of six interleaved pairs.
* Say that bool is in, and what its first stage found (016)eunhhu2026-09-301-1/+19
| | | | | | | 016's first stage is built and checked: no opcode was new, a stored send's bool was refused on the way back in (found by server_meaning's restart, fixed), -0 becomes false as 015 says zero has no sign, and True in JSON is no JSON at all. i64 is what is left of 016.
* Check a bool view's text and a bool argument over HTTPeunhhu2026-09-301-0/+1
| | | | | | | server_meaning's origin gains a bool view and a view with a bool argument: GET /views/any_ok answers true as text, with_ok?want=true and ?want=false count the kept values each way, and 1, True or nothing for the argument is 400 invalid.
* Pin a bool's meaning on every path (015)eunhhu2026-09-301-4/+20
| | | | | | | | | | | | | | | server_meaning gives each of its 67 values a bool field, true or false: in as JSON, a query, a message's rows; over the link to B (taking it by name among reordered fields) and over HTTP to C; out through A's view and query, B's and C's views, a restart, a restart after a checkpoint and B with its module replaced. A bool must come out as true or false, never 1 or 0. Refused on every way in, with the reason checked: 1, 0, "true", null or an array as json_type, True as json (no JSON at all), a query's 1, 0, True, TRUE, "true " or nothing as query, a row's 1 as json_type: 52 refusals, 39 before. 015 says what a bool is at each boundary. This check found the stored outbox refusing bool fields (a node could not start again with a bool send waiting), fixed in the commit before.
* Give bools their own type in the module format (v6, 016 stage 1)eunhhu2026-09-302-46/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A condition was a number, non-zero being true: a flag was a number field, a view showed 1 for true, and nothing kept a number from standing for a bool or a bool from being added to. Now the verifier has a bool among its stack types, and a verified module never mixes the two. No opcode is new: - CONST with argument 1 is a bool (+0 or 1), NUM_TEXT with argument 1 writes one as true or false; LOAD and STORE take a number or a bool state, locals may hold a bool (2). - Comparisons (LT to NE, TEXT_COMPARE) and LIST_HAS give a bool; AND, OR and NOT take and give them; JZ, REQUIRE, filters and folds' tests take one (count and first leave a bool, sum, max, min and distinct a number and a bool); EQ and NE compare two bools too. - Type codes: state 5, record field 2, message field 6 (one byte in the prefix, 0 or 1, or the message fails as limit), view 3. A bool's constant, initial value and default are +0 or 1; a list is not keyed by one. PUREMOD6: v5 modules are built again. At run time a bool is a number's 0 or 1, so leaf-at-a-time folds run as before. A number that becomes a bool (a module replaced, a stored state or item read again, a waiting send recast) keeps 0 (either sign) and 1 and takes the default otherwise; a bool that becomes a number is 0 or 1. The host carries bools on every boundary: JSON true and false only (a row's field too), a query's true and false, true and false in views, table JSON and HTTP deliveries; a link takes a bool only into a bool. A stored outbox's field types were read back with a check that refused bool (at or past rows): a node whose output had a bool field and a send waiting could not start again. test-module now stores such a send and opens it (it fails on the old check), beside 28 bool refusals of the verifier and bool states, fields, views, sends, links and migration. The lane generator makes numbers and bools apart.
* Say what bool and i64 mean at every boundary, before building them (016)eunhhu2026-09-301-0/+74
|
* Say what the screen showed of a value, and what Vega now keeps from acting ↵eunhhu2026-09-301-1/+13
| | | | (015, 108)
* Say what a value means at each boundary, and what checking it found (015, 108)eunhhu2026-09-301-0/+70
|
* Say what laying a record out anew costs (014)eunhhu2026-09-301-1/+1
| | | | | | | A board record of five fields (38 B) sent with v3's output fields took 310 ns to lay out for v4's input, reading the field list each time. Only streams of another shape pay it, and only while nodes are upgraded one after the other: less than a delivery's own work.
* Take a delivery's fields by name, dropping noneeunhhu2026-09-302-4/+93
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A message from another node was taken only in its input's exact shape (the link's STREAM shape, fields and order alike; over HTTP, every field). So a board whose nodes are upgraded one after the other stopped both ways until both were, and an output and its input whose fields differ only in order were refused for good (docs/design/014). An input now takes a message when it has every field the sender sends, by name, of the same type or one that holds its values (a u32 in a number, text in bytes); a field only the input has is 0 or empty. One that sends a field the input lacks is refused: taking it would drop the value, and with its number heard the value would never come again (two boards apart for good). So add fields to receivers first and take them from senders first, and nothing stops or is lost. The link carries a sender's fields in a new frame, FIELDS (0x0d: names, shape, u8 count, each {type, name length, name}): after a REFUSED for another shape the sender puts it in its packets (after OFFER, before the records, so a receiver that knows no FIELDS reads what it read before) until the stream is taken, the first time at once. A receiver seeing another shape waits for the packet's FIELDS before refusing, and asks its host with them (PureLinkHost.input takes them); REFUSED says 4 when the input lacks a field. Deliveries carry the fields and the host lays each record out anew (pure_module_takes, pure_module_recast, pure_module_fields, over pure_send_recast's rule). A stream whose sender starts again in another shape lets go what it held; the sender sends it again laid out anew. A refusal already queued for a stream now says the last reason judged, not the first. Over HTTP a delivery (X-Pure-Source) may lack fields; an unknown member is still refused. test-link named: a stream of another shape naming its fields is taken, 200 stored in 47 ms with every delivery naming them; an input that does not take them refuses (4) and the sender backs off. check.py server_fields: five outputs, three linked and two over HTTP, fields in another order taken, an input with a field more taking messages without it, one lacking a field refusing until it starts again with it, then taking all. server_upgrade's new receiver declares its fields in another order. 012 names the frame and the rules.
* Say what the windows and checkpoints cost (013 steps 2 and 3, 104)eunhhu2026-09-301-6/+16
| | | | | | | | | | | | | | | | | | 013's second step is in (d9e0b68): each transport holds a window of its output's sends. 104 measures it with the receivers down: once the window is full a node grows by the outbox alone, 219 and 218 B a send over the link and HTTP (399 and 534 B before). The third step is a measurement. A checkpoint holds one list's snapshot while it writes the image, so what it adds is what the app changed meanwhile: nothing visible on tmpfs or for a list that fits the card's fast writes; on the SD card, a 22.4 MB list written for about 8 s while 1,400 items a second were set left +3.0 and +4.4 MiB. The highest resident memory was twice the state bytes, inside 013's bound (the state and one snapshot) and its stop criterion (3x). 021's run.py takes --items, --stores (a directory on the device to measure) and --label, and reports the peak (VmHWM).
* Say where a node's memory goes, and the budget it now keeps (104, 013)eunhhu2026-09-302-0/+99
| | | | | | | | | | | | | | | | | | 104 measures a node's memory under a receiver that is down and under a snapshot held for long (021's run.py: resident memory against what was posted, 30 s a case). A held snapshot grew the node without end (6.4 to 678 MiB, fixed in d1e8f38); a slow receiver grows it by what was sent, 221 B a send, which is state until delivered, so the budget bounds state and refuses past it. 013 is that budget's design: what it counts (state bytes, deterministic), what an event over it does, why an answer takes its send's room, how links and HTTP wait, what it does not count and those bounds, and the checks. With --memory 4000000 the same load fills it in about 4 s and the node stays at 9.6-9.8 MiB (45.9 MiB and growing without). 012 records the hole the budget found in the link (the full flag), and 103 that its clock fix fell short (385754c). CHECKPOINT gains the row.
generated by cgit v1.2.3 (git 2.46.0) at 2026-10-20 22:18:15 +0000