| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
| |
walk's spread items (PLAN_FILTER; L only grown there, one filter a list, none in a nested LOOP): each round asked as the batch opens, its code run in the rounds it said no to, those rounds kept (k and the appends' cells) and asked again at the end in the rounds' order (L as it is and what the kept rounds before add), the rest added at once (pure_list_add_many) once the lists take them and the memory budget all of them (else, or a key added there already, the batch gives its rounds back); a list such a filter grows read in the block only as no earlier round's item would change (its count in the guard, its first, a place when it has no key, a field by key). Keys kept as bits (pure_list_bits) asked by a loop of lanes, fields of a DENSE list found at their keys (pure_list_dense); an outer lane's affine value in four bytes. The suite's BFS batched: PURE_JIT=0 93 to 39.6 ms (the compiled code 7.7, C 10). module_tier's walks filter d (seeds 1, 2, 3, 5, 23 alike: 5,806 filters decided in seed 23's); GRAPH_SOURCE: a graph built by a filter, searched twice, marks whose filter adds another key (EXISTS), under work and memory budgets
|
| |
|
|
| |
only grown in it) batches to L's count as each batch opens (PLAN_GUARD; fewer than LEAST rounds left there: the executor's, then batches again); a walk in a batch's block (its slice per lane, no where's range) runs each lane's items, its work counted as each opens: together round by round (BREAK as a nested loop's), or spread into lanes in the rounds' order a chunk of up to 1024 at a time when its items' rounds may run apart (PLAN_SPREAD: no break, the locals it sets reductions or its items' own: an outer lane's value read by each item's outer lane, a reduction added into its outer lane's in the items' order, an own local left as the last item that set it); rows in four bytes read and checked by loops of their own. A frontier walk of the suite's graph (each node's edges' to summed) PURE_JIT=0 55.2 to 10.4 ms (the compiled code 6.24). module_tier makes guarded loops with a walk in each (seeds 1, 2, 3, 5, 23 alike: 8,819 spread and 7,651 together in seed 23's); FRONT_SOURCE
|
| |
|
|
| |
128) when every list takes its items so (an event's list, no index into it, one append to it: FLAT not keyed at its end, DENSE by keys it holds at its count, HASHED by number; never past its arrays' limit) and the memory budget takes them all (each item only grows what it counts): columns grown and widened once, the first key there already the executor's EXISTS at its round; else round by round as before, tried at once again after a round that changed a list's form. 50,000 DENSE adds 5.18 to 0.65 ms (the compiled code 0.28). APPEND_SOURCE: keys there already at rounds q and p, keys past a DENSE list's from round p, two appends to one list, widened columns, under memory budgets (seeds 1, 2, 3, 5, 23 alike: 31,402 batches at once in seed 23's, 292 by the budget round by round)
|
| |
|
|
| |
at a time: each lane an item's place (ROW of the walked list affine), its work counted as it opened, its place moved past the batch; no append to the walked list. module_tier makes walks of rounds apart (f, g, h, slices of g; seeds 1, 2, 3, 5, 23 alike: 1,891 walk batches in seed 1); BATCH_SOURCE walks a slice
|
| |
|
|
| |
e, neither reading L; the set at k's row as found), keeps its rows and e from the pass (the second k, its find and the cell's read skipped) and adds them at the batch's end round by round as the executor adds and sets (a sum not finite: the executor's fault at its round; a column too narrow widened); a batch reading or setting those cells otherwise gives its rounds back. Recurrence steps one dispatch each. PURE_JIT=0 join 428 to 165 ms (the compiled code 78). module_tier makes sums into cells (seeds 1, 23 alike: 278 batches with them in seed 1); BATCH_SOURCE sums into three cells in turn
|
| |
|
|
| |
sets it, every set of it x = e of x, numbers, fields, states, the loop's value and locals the block does not set, in no branch's way; its sets planned PLAN_RECUR) runs ahead of the batch round by round as steps decoded once a batch, as the executor computes them (a round's fault gives the batch back), leaving its value at each other read in each lane; the batch's pass skips its sets. matmul's, knapsack's and join's table loops batch: matmul 50 to 46 ms. module_tier makes loops with a recurrence (x1) ahead of appends, sets and sums (seeds 1, 2, 3, 5, 23 alike); RECUR_SOURCE: two updates a round, a division by 0 and a product past the numbers at some round, under work and memory budgets
|
| |
|
|
| |
appended in a batch's block (to a list the block reads nowhere else, no index into it, no nested loop) is made at the batch's end round by round through the executor's own step: keys there already, the memory budget, its undo log as the executor's; a fault there is the executor's at that round (the event undone: its kind, its instruction and the work so far tell), pure_batch_run's third way out. module_tier's handlers append (e, d) in rounds apart and run under a memory budget too (seeds 1, 2, 3, 5, 23 alike); knapsack's table loop batched (13.8 to 13.7 ms: the appends cost as the executor's)
|
| |
|
|
| |
for every lane together (round t for the lanes with more than t rounds that did not break out; BREAK leaves it lane by lane; its slot as each round sets it), and x = x op e (op +, min, max; x read no other way) is a reduction: each lane's own from op's identity (-0 a sum's), folded in the rounds' order at the batch's end (a sum's lanes of several whole numbers only, below 2^53); nested rounds' work counted for each lane and checked before anything changes; lanes and masks no holder keeps freed between nested rounds. Planned at load: a local read before set in its loop's scope, or not a reduction's, keeps the loop round by round. PURE_JIT=0: roots 130.5 to 16.8 ms, matmul 914.7 to 68.2. module_tier makes nested loops with breaks and reductions (seeds 1, 2, 3, 23: 549 programs alike); NESTED_SOURCE
|
| |
|
|
| |
(lane forms, locals set before read in every round, ifs, numbers set in items) runs, in the executor, up to 1024 rounds at a time: values uniform, affine or lanes (numbers, or whole numbers in four bytes), branches as masks, cells read by affine rows as strided columns, sets kept to the batch's end and made only when no round reads an earlier round's; a fault, a form it lacks or such a read gives the rounds back to the executor one by one, work counted as written. Planned at load (PURE_BATCH=0: none); the executor of a module without such loops is the same code (run aligned to 64: as many instructions and cycles), with one its own copy. knapsack with PURE_JIT=0: 1,056 to 13.7 ms. module_tier compares batches with the executor alone and makes loops of rounds apart; BATCH_SOURCE: widened columns, -0, a fault mid-batch, rounds reading the last's sets
|
| |
|
|
|
|
| |
round, in range for every round (its opening found it) and in a full list, reads its cell through a pointer, a phi at the loop's head: the column (read from the list) plus the key at round 0 times four where the loop opens, its key's step a round times four added at its latch. The key's nodes, in range, give up nowhere (no snapshot kept), so a key read by such accesses alone is not made; a list found by key alone with such an access is full where the tier begins (else the generic code runs the nest). matmul 23.0 to 14.2 ms (3.1x to 1.9x C)
module_tier's programs sweep lists by keys stepping with a round (in range or not, a list full or not)
|
| |
|
|
|
|
| |
(constants between), checks only the last is finite, giving up at the run's first snapshot (the executor makes the run again and fails where it does); a snapshot's constants written as the constants they are, not kept in registers; the generic code's append in a loop keeping its books puts into a HASHED list as append does (its headroom kept). roots 12.2 to 10.9 ms (1.59x C), join 84 to 74 (2.5x)
module_assumptions: a run of products and sums past the largest number (CHAIN_SOURCE)
|
| |
|
|
|
|
| |
each item's cells at its key's slot in its members' table, found by key there (the key and its cells read at one slot, as C's table); its order made by a radix sort of the slots' keys when something asks for a position (twice an event, then FLAT or a tree). The tier finds by key alone in a HASHED list (its bits' bit 1, its limit all ones: the bounds checked at a loop's opening still hold) out of the line; a list keyed by each count in order is DENSE from its first item. join 531 to 108 ms (3.5x C), sort 98 to 58 (2.1x)
arrays_equivalence counts the HASHED form against a tree; module_tier's programs read a HASHED list by key
|
| |
|
|
|
|
| |
rounds it does not run given back where it breaks or gives up), a comparison only a branch reads its flags, a full list's keys read without its bits (a key's position found: its key's register), an add's or sub's register its operand's where that is read last. knapsack 16.5 to 8.5 ms (1.37x C), matmul 25.7 to 22.9
Two bugs the work checks found, there since 6cd7f5a: the outermost loop's round past the work budget gave up at the loop's instruction, with nothing on the stack (the executor opened it again: LIMIT where WORK), now at its iteration; and a phi where an if's ways meet took its merge's loop (-1), so a value made of it was taken as made outside its loop and made once for all rounds. pure-run's budget and work commands; module_tier compares the work and runs four small budgets; module_assumptions keeps both programs
|
| |
|
|
| |
values in registers (x20-x28 too, the frame's read again where it ends), whole numbers integers by their facts, the message's fields and lists' columns read once, invariants made before inner loops; where anything does not hold it gives up to the executor at the instruction (pure_run_rest) and the generic code goes on. roots 22.9 to 14.0 ms, knapsack 1,030 to 38.8, matmul 908 to 48.6; module_tier checks random handlers against the executor
|
| |
|
|
| |
small constants and powers of two's products in the instruction; where a check may give up, make the result in x16, the operands as they were for the executor (aup's check caught a register shared with its operand): BFS core 12.5 to 12.2 ms, 46.2M to 43.0M instructions (022)
|
| |
|
|
| |
(narrow columns, counts, their arithmetic within 2^53), giving up to the executor for the rest of a round when that fails: BFS core 14.9 to 13.4 ms (022)
|
| |
|
|
| |
meanwhile, and queue=N's busy and its places coming back (020)
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
and pushed (020)
- pure-serve --view-workers N (a CPU each; 0: on the connection threads):
a GET of a view with a body (or a table with such a column) goes to the
workers, which show it from the app's newest snapshot on readers of
their own and give the answer back as an owner does (eventfd). The
connection thread reads and writes its other connections meanwhile.
- A snapshot of another module than the request was resolved with, or
none (the app stopped), answers 503 stopped. Past an app's queue=N of
them, 503 busy.
- A push group of such a view hands one evaluation at a time; a newer
snapshot meanwhile is evaluated after it; a group whose subscribers all
left stays until its evaluation is back.
- check.py server_view_workers: 8 askers of a long view while its app
changes module (another view in its place) and then goes: every answer
the old module's, the new one's, 503 or 404; streams closed at once
leave nothing; --view-workers 0 as before.
|
| |
|
|
| |
show a subscriber costs writes nothing (019)
|
| |
|
|
| |
(019)
|
| | |
|
| |
|
|
| |
bounded (018 stage 2)
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
An app given a tokens file (tokens=FILE in --apps, --tokens FILE for the
one app of a host) answers a request only when it bears a token
(Authorization: Bearer) whose SHA-256 the file lists with a grant: read
for its views, queries and documents, write for all, input=NAME for
those inputs alone; 401 unauthorized (WWW-Authenticate: Bearer) without
one, 403 forbidden past its grant. A delivery sealed by a key --sender
names needs none. The digests are compared in constant time; a
connection keeps its last token's digest, so the same token is hashed
once. SHA-256 is the host's own (serve/sha256.c), no OpenSSL loaded. A
reading puts a changed tokens file in place with the app running on; a
file that is not one stops the app. pure-serve --make-token GRANT makes
a token and its line. load takes PURE_LOAD_TOKEN.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Stopping an app and starting it again reads its whole store again: a
million items took 142 ms, as long as restarting the host (030). A
running app whose line changed but not its store now goes quiet (what it
took applied, saved and answered, its batches back, its outlets closed,
its checkpoint finished), has its module replaced in memory
(pure_app_replace carries its states, and the sends in its outbox, by
name), its budgets set and its outlets made anew, and resumes; a module
its states do not go into stops it. An app whose store changed, one a
save of which is uncertain, and one that does not run still start from
their stores. A reading says changed for the first. A snapshot is
published again when the module changed though the states did not.
server_apps_lifecycle reads the changing apps' views (lists among them)
as they change, and moves an app's store; it passed 10 times alone, 5
under load and 6 with an ASan pure-serve.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A host of apps reads its --apps file again on SIGHUP, on a thread of its
own (pure-apps, the one that takes the signal): a file with a wrong line
changes nothing; the apps of lines gone stop and their routes go (404
no_app); those of changed lines, and those not running (they could not
start, or a save of theirs is uncertain), start again from their stores,
states carried by name; new ones take places of their own; a line out
says what was done. An app stops by taking nothing more (503 stopped),
applying, saving and answering what it took, waiting for its batches at
other apps, then closing its outlets (what they did not deliver stays in
the outbox) and its store. Its link streams find their inputs again as it
starts. Owners' locks are made with their places; counts are kept over
an app's runs; workers grow with the apps that run, a CPU each at most.
server_apps_lifecycle adds, removes and changes apps, and changes the
receivers (a local outlet's and a link's, their inputs at other numbers,
taken by name) and the senders while 600 sends each and writes go: every
send arrives once and in order, a write answered 200 is applied once and
one answered 503 stopped not at all, the app left alone fails no request;
an app whose store cannot be made starts at the next reading once it
can; kill -9 and the host starts as the file says. It passed 10 times
alone, 5 under load, 6 with an ASan pure-serve.
|
| |
|
|
|
|
|
|
|
| |
server_timers compares the times an app saw with the wall clock read
around them. WSL2's clock is set back about 2 s every 32 s (research
029): a window holding such a step says nothing of the times seen, and
failed the check there 4 times in 20. It is taken again (up to three
times) when the wall clock and the monotonic one moved more than 50 ms
apart over it; the report counts the windows taken again.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
send=OUTPUT=APP/INPUT in --apps wires an output to another app of the
same host through a local outlet: no socket, seal or packet. The
sender's turn queues its saved sends there, under the window links
and wires have, and hands them, one batch at a time, to the receiving
app's queue laid out as a link delivers records (first number, count,
source APP.OUTPUT, the records as stored, and the sender's fields when
the input takes them by name, which pure_module_takes checks at start).
The receiver applies them after its last number heard; its save over,
it tells the outlet what it heard and the sender's next turn trims its
outbox so far. What it did not store is queued again from the outbox,
after 100 ms when it had no room. An output now goes one way only: two
outlets would trim each other's sends.
server_apps_links: an app sends each of 400 messages to an app of
another host over a link and to three apps of its own host (one of
the same shape, one taking the fields by name, one with a memory
budget that fills), with a kill -9 of the sending host between two
requests; all arrive once and in order, and the full app's sends wait
in the sender's outbox until it lets go of what it holds.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
An app of --apps takes send=OUTPUT=URL, seal=OUTPUT=KEY and
reply=OUTPUT=INPUT, as --send, --seal and --reply give the one app of a
host; its sends' source is APP.OUTPUT. An app wired wrong stops alone,
its wires closed. The host's --key, --sender, --senders, --sealed-only
and --link-port now go with --apps: the one listener resolves a
stream's input name APP/INPUT to that app's input (a stopped app or an
unknown name is no input, REFUSED 1), its deliveries queued at that
app's owner. 012 is unchanged.
server_apps_links: an app of one host sends 300 messages, sealed, over
a link to an app of another host, which takes them all once and in
order.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With several connection threads, a host of apps had each app's
connections spread over all of them: every thread's round handed each
owner a share, and every turn handed answers back to every thread. The
apps took a third more CPU a request than as processes, with five times
the voluntary switches, and when two apps were busy each waited a
kernel tick at its 99th percentile (4.1 ms, against 1.5 ms as two
processes).
Each app now has a home thread (its place in --apps, modulo the
threads). A connection whose first request names an app at home
elsewhere moves there once, its head read and no ticket taken yet: the
home thread adopts its descriptor and the bytes read, and reads the
request again. Each app's requests then meet its owner from one
thread, as from a process of its own. Two busy apps: 99th percentile
1.4 ms, the same 145k writes a second.
server_apps restarts its apps on three connection threads, asks each
app from fresh connections, and checks that some moved and all were
answered right.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pure-serve --apps FILE hosts an app a line: NAME MODULE STORE and its
budgets (memory=, work=, queue=). Each app's routes start with its name
and reach its own module, states, snapshot and queries; a view is shown
on the connection thread from its own app's snapshot, read once a round.
No app has a thread of its own. Its owner is what used to be a thread's
loop, now a turn: its due timers, what its wires delivered, and the
requests waiting for it applied and saved once, its answers handed back.
Workers (--workers N; one for a host of one app, else one a CPU at most)
take apps from a queue in turn; a worker sleeps until the soonest timer.
A turn takes every waiting request when no other app waits, else 256;
it gathers the clients its last save answered only when alone.
An app's queue budget refuses what would wait past it (503 busy) on the
connection thread, before its owner sees it. An app whose module does
not load, whose store does not open, or whose store is another app's
stops alone: the start line says why, and its routes answer 503
stopped. The stop line counts each app's turns, events and refusals.
A host of one app (--module) runs as before, its owner on one worker
named pure-owner: every check passes unchanged. server_apps runs six
apps of which two stop, floods one past its queue while another
answers, refuses a walk past one app's work budget that another runs,
holds an app's store from another process (that app answers 503
store_busy and applies nothing, the others save), and restarts.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
An event's run was bounded only by its lists: a walk or a fold is its
block's length for each item, and a list holds up to 16 million. One
event could hold the owner for seconds, one view a connection thread.
pure_app_work(app, n) budgets one unit of work: an event, or one
evaluation of a view, a cell or a table's filter. It is counted as
written, as each part begins: a block's length, and a fold's or walk's
block length for every item of its list (or slice), whatever the plan
runs (lanes, a group or rank pass, a range). Nothing is counted inside
the interpreter's loop. Past the budget an event fails with the new
fault work and changes nothing (PURE_LIMIT, so 422 work over HTTP); a
view, a cell or a table's rows have no value and pure_app_view_fault
says work, which pure-serve answers 422 work, for a whole table when
a cell of it ran past. pure-serve --work N sets it (0: none).
test-module counts exact amounts on the shop's walk, the tasks' fold
view and a filtered table, and lane_folds now checks that a planned
fold counts what the same fold run item by item counts (3,600 folds,
every view, cell, filter and handler). server_work grows a list over
HTTP until the walk, the table and the fold each pass a budget of 400.
|
| |
|
|
|
|
|
|
|
| |
server_meaning gives every value an i64 field, 13 values from 0 and -0
to both ends of the range and just inside them, in over JSON, rows and
a query, over the link (where B takes id, a u32, as an i64) and HTTP,
and out at all seven exits with every digit. 32 more refusals, each
with its reason on JSON, a query and rows; an i64 view's text at both
ends and an i64 argument that counts exactly and refuses the rest.
|
| |
|
|
|
|
|
| |
server_meaning's origin gains a bool view and a view with a bool
argument: GET /views/any_ok answers true as text, with_ok?want=true and
?want=false count the kept values each way, and 1, True or nothing for
the argument is 400 invalid.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
server_meaning gives each of its 67 values a bool field, true or false:
in as JSON, a query, a message's rows; over the link to B (taking it by
name among reordered fields) and over HTTP to C; out through A's view and
query, B's and C's views, a restart, a restart after a checkpoint and B
with its module replaced. A bool must come out as true or false, never 1
or 0. Refused on every way in, with the reason checked: 1, 0, "true",
null or an array as json_type, True as json (no JSON at all), a query's
1, 0, True, TRUE, "true " or nothing as query, a row's 1 as json_type:
52 refusals, 39 before. 015 says what a bool is at each boundary.
This check found the stored outbox refusing bool fields (a node could not
start again with a bool send waiting), fixed in the commit before.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A condition was a number, non-zero being true: a flag was a number field,
a view showed 1 for true, and nothing kept a number from standing for a
bool or a bool from being added to. Now the verifier has a bool among its
stack types, and a verified module never mixes the two. No opcode is new:
- CONST with argument 1 is a bool (+0 or 1), NUM_TEXT with argument 1
writes one as true or false; LOAD and STORE take a number or a bool
state, locals may hold a bool (2).
- Comparisons (LT to NE, TEXT_COMPARE) and LIST_HAS give a bool; AND, OR
and NOT take and give them; JZ, REQUIRE, filters and folds' tests take
one (count and first leave a bool, sum, max, min and distinct a number
and a bool); EQ and NE compare two bools too.
- Type codes: state 5, record field 2, message field 6 (one byte in the
prefix, 0 or 1, or the message fails as limit), view 3. A bool's
constant, initial value and default are +0 or 1; a list is not keyed by
one. PUREMOD6: v5 modules are built again.
At run time a bool is a number's 0 or 1, so leaf-at-a-time folds run as
before. A number that becomes a bool (a module replaced, a stored state
or item read again, a waiting send recast) keeps 0 (either sign) and 1
and takes the default otherwise; a bool that becomes a number is 0 or 1.
The host carries bools on every boundary: JSON true and false only (a
row's field too), a query's true and false, true and false in views,
table JSON and HTTP deliveries; a link takes a bool only into a bool.
A stored outbox's field types were read back with a check that refused
bool (at or past rows): a node whose output had a bool field and a send
waiting could not start again. test-module now stores such a send and
opens it (it fails on the old check), beside 28 bool refusals of the
verifier and bool states, fields, views, sends, links and migration.
The lane generator makes numbers and bools apart.
|
| |
|
|
|
|
|
|
|
|
| |
Values at the edges of what JSON, rows, a query and a plain body can say
go in to one node, over a link to a second that takes them by name in
another order and wider, and over HTTP to a third; each must come out
the same from views, a query, an answer, a restart, a restart after a
checkpoint and a replaced module: a number's bits (zero without its
sign), a text's bytes. What must be refused is refused on every way in
and leaves nothing behind.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A message from another node was taken only in its input's exact shape
(the link's STREAM shape, fields and order alike; over HTTP, every
field). So a board whose nodes are upgraded one after the other stopped
both ways until both were, and an output and its input whose fields
differ only in order were refused for good (docs/design/014).
An input now takes a message when it has every field the sender sends,
by name, of the same type or one that holds its values (a u32 in a
number, text in bytes); a field only the input has is 0 or empty. One
that sends a field the input lacks is refused: taking it would drop the
value, and with its number heard the value would never come again (two
boards apart for good). So add fields to receivers first and take them
from senders first, and nothing stops or is lost.
The link carries a sender's fields in a new frame, FIELDS (0x0d: names,
shape, u8 count, each {type, name length, name}): after a REFUSED for
another shape the sender puts it in its packets (after OFFER, before
the records, so a receiver that knows no FIELDS reads what it read
before) until the stream is taken, the first time at once. A receiver
seeing another shape waits for the packet's FIELDS before refusing, and
asks its host with them (PureLinkHost.input takes them); REFUSED says 4
when the input lacks a field. Deliveries carry the fields and the host
lays each record out anew (pure_module_takes, pure_module_recast,
pure_module_fields, over pure_send_recast's rule). A stream whose sender
starts again in another shape lets go what it held; the sender sends it
again laid out anew. A refusal already queued for a stream now says the
last reason judged, not the first. Over HTTP a delivery (X-Pure-Source)
may lack fields; an unknown member is still refused.
test-link named: a stream of another shape naming its fields is taken,
200 stored in 47 ms with every delivery naming them; an input that does
not take them refuses (4) and the sender backs off. check.py
server_fields: five outputs, three linked and two over HTTP, fields in
another order taken, an input with a field more taking messages without
it, one lacking a field refusing until it starts again with it, then
taking all. server_upgrade's new receiver declares its fields in another
order. 012 names the frame and the rules.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Review 004's F3 made a module refuse to start while sends waited on an
output whose fields it had changed: their bytes would have meant
something else. Safe, but it deadlocks a rolling upgrade. Growing a
board shared by two nodes: one upgraded to a card with a new field
refuses the other's old sends (another shape), and the other, its old
sends waiting, cannot start the new module at all ("Cannot start the
module on the store", saying nothing more).
Waiting sends now move to the new fields by name, as a list's items do:
a value goes over when the new type can hold it (u32 and number are
numbers, bytes and text are bytes; a number that is not a u32, or bytes
that are not UTF-8 for a text, cannot), the rest start at 0 or empty
(pure_send_recast, with the payload layout load uses, pure_fields_lay_out).
A start recasts them and a replace does into a new outbox.
Their meaning must survive a crash between the recast and the save that
names the new fields in the links. So the outbox's image now names the
fields its sends were laid out with ("PLO1": the module's outputs after
the save it holds), and is written again before any send is logged with
other fields; a start reads those names over the links'. Checkpoints of
the outbox name them too.
A module without an output that sends wait on is still refused, now
saying so in PureApp.why, which pure-serve prints ("sends wait on the
output 'ordered', which this module does not have"); a failed start
keeps it.
test-module send_recast (widened, narrowed, UTF-8, a payload not laid
out, there and back) and output_meaning (the swapped fields recast on a
replace, read right after a restart that never saved, back again, sends
logged after the image, the refusal and its reason). check.py
server_upgrade: five sends wait on the old output, the sender starts
with the new one, the old receiver refuses the new shape, the new one
takes all eight.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A link's ring and a wire's queue took every saved send of their output
and kept it until delivered: with the receiver down they held the
outbox a second time, 178 B and 313 B more a send (104). Each outlet now
holds a window, --window bytes (4 MiB) and 65,536 sends past what its
receiver has, at least one; the outbox keeps the rest.
The owner remembers each outlet's place in the outbox and looks on from
there after a save and after its transports deliver. Places move back by
the sends that left the outbox since (PureApp.sends_gone), so a trim
before a place skips nothing, and sends queued before are not looked at
again however many wait. pure_app_sent reads a send by its place (the
newest-first visit it replaces is gone); pure_link_held and
pure_wire_held say what a transport holds.
With the receiver down a node now grows by the outbox alone once the
window is full: 219 and 218 B a send over the link and HTTP (399 and
534 B before). check.py server_windows keeps the receivers of a sender's
two outputs, one linked and one over HTTP, down through 300 buys with a
2,000-byte window, then starts them: each applies all 300 once. Without
the refill after deliveries the link stops at 117.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Review 004's timer fix (103) counted the deadlines on the monotonic
clock, but the owner still slept until a wall-clock time: with the
clock set back an hour a node's timers would stop for that hour. So
would a wire's pause between tries, and a lone commit after a group
would wait an hour for the others to join it. The test's LD_PRELOAD
moves only what the process reads, so there each wait ended at once
instead, and the timers went on ticking while the owner spun a core.
core/wait.h makes conditions that wait on the monotonic clock
(pthread_condattr_setclock; macOS keeps the wall clock) and deadlines
on it. The owner's waits, the wire's pauses and the group commit's
gathering use them.
clock-back.so can now move the clock forward too (CLOCK_SHIFT_S), and
check.py server_clocks moves it an hour each way under a node with a
timer and a wire to a port no one listens on, and under a document
server whose three threads commit together, then one alone. Before:
back, 159 % CPU and 50,886 tries in 1.2 s; forward, 1 tick, 1 try, and
a lone commit that never ended. After: 59 ticks, 0 % CPU, 4 tries and a
lone commit of 2.6 ms, both ways.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pure-serve --memory BYTES gives the app a budget of state bytes: its
lists' items as they are stored, its texts, the outbox and the numbers
heard (docs/design/013-memory-budget.md). The count is deterministic,
the same on any device. An event after which the states hold more than
the budget, and more than before it, fails with the fault memory and
changes nothing; HTTP answers 503 memory. One that frees or keeps them
goes on, so an app over its budget can always shrink.
Sends waiting for their receivers are state: a full outbox refuses the
next send until some are stored, which is the backpressure a slow
receiver puts on clients. A delivery refused for memory is not heard:
the link keeps it (c9c0edf) and HTTP senders try again after 503.
An answer (--reply) now takes its send out of the outbox in its own
event (pure_app_answer), so it has the room its send held. Apart, an app
whose outbox filled the budget could never apply an answer that keeps
anything, and the send leaves only after its answer: both would stay for
good. An answer that still does not fit waits, with the wire's after it,
and is tried again every 100 ms: pure_wire_answer leaves it in place
until pure_wire_answered.
Checks: test-module memory_budget (refused and undone, freeing over the
budget, a full outbox, a delivery not heard, answers that fit or wait,
across a restart); check.py server_memory (a sender refused 503 memory
with its receiver down and taking buys once it is up; a receiver
restarted below what it keeps, refusing 7 times in 2 s, then taking all)
and server_answers_memory (answers wait while receipts fill the budget,
then apply once each; the API called once per order). The stats report
state_bytes and memory_refused.
|
| |
|
|
|
|
|
|
|
| |
The usage showed --send OUTPUT=http[s]:// first; links are now the way
between nodes (012), so it shows pure:// for a Pure node and HTTP for
anything else. The statistics at exit count the streams a node's links
were refused (link_refused) and the ones its listener refused
(listener_refused), so a wiring mistake is in the numbers too, not
only the line on stderr; server_links_refused checks both.
|
| |
|
|
|
|
|
|
|
|
| |
pure-serve took a module with an input named show or edit for a
document app and refused it unless both were keyed messages: an app with
a plain message named edit compiled and then would not start (review
004, an earlier P2). Documents are what a module keeps in a keyed
collection; now the host serves them when the module declares one, and
inputs named show and edit are otherwise messages as any other.
check.py's server_plain_edit starts such an app and posts to edit.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
run_timers ticked nothing in an app without states, taking a timer to
be able to change nothing else; but a tick may send, and the outbox is
the app's state (the host gives such an app an owner for it). A timer
that only sends never ticked (review 004, F4). Now an app ticks when it
has states or outputs; one with neither runs on every connection thread
and has nothing a tick could do.
When a timer is due was reckoned on the wall clock: set back, a timer
waited for the time it had reached again (review 004, an earlier P2).
The next tick is now reckoned on the monotonic clock; events still see
the wall clock (now).
check.py's server_timers runs the review's heartbeat (a timer, an
output, no state) and sees it tick, then again with the wall clock set
back an hour 0.2 s in (build/clock-back.so, an LD_PRELOAD of
clock_gettime): 29 ticks in 0.6 s, where the server before this stops at
9.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A walk whose where bounds a keyed list's key finds the bound once, when
the walk begins, if its body writes nothing the bound reads. The body's
writes counted only states stored and lists changed, not what those
changes move with them: an insert or remove moves the indices into the
list, a text's change the cursors in it. A walk bounded by an index
into a list its body inserts into saw the bound it began with: of five
rows it took one (review 004, F1). Those states now count as written,
and a bound that moves an index (whose list is whichever the index's
is) is kept only when the body changes no list.
build/module-probe runs a module twice from the same keys, as it loads
and with every range let go, and says which state first differs.
check.py's module_ranges runs the review's case (all rows twice: done
10, at 10) and a walk whose bound its body cannot move, still ranged.
The probe finds the difference in the runtime before this.
|
| |
|
|
|
|
|
|
|
|
| |
check.py's server_links_independent is a receiver in Python (the
cryptography package, no code of link/) written from the link protocol's
description: the static secret, HKDF's keys, the nonces, the frames, a
CHALLENGE, ACK and STORED, and ACCEPT for epoch 1. A pure-serve node
links to it: every one of 200 orders is stored as the node knows it,
the first packet opened in epoch 0 and the other 199 in the epoch the
two made apart. None where the package is not there.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A receiver that took no records of a stream (a source the key may not
send as, no input of that name, an input of another shape) said nothing:
its sender heard them acknowledged, never stored, and sent them all
again every second, for ever, and its operator saw nothing. Now the
receiver answers REFUSED (0x0c: the names, and why), and the sender
sends no records for a second, then a packet of them to learn whether
they are taken now, the wait twice as long each time (64 s at most). A
refusal ends when that packet is acknowledged with no REFUSED beside, or
a record is stored. link.c says it once on stderr, as a wire says a
receiver does not take its sends; serve's input says which of the two
the app lacks.
test-link: each of the three is refused seven times in 90 s with nine
packets, and a stream refused until the app has its input goes on then,
all 200 records stored. check.py links a node to an input its receiver
lacks: said once, two packets in 2.5 s.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A link resolved its receiver's name once, at its start: a receiver whose
store moved to another machine (another address under the same name)
was never found, and a sender whose own network changed kept a socket
that no longer went anywhere. Now, when three probes in a row go
unanswered, and at each probe after (a second apart at most), the link
looks the name up again and sends from a new socket connected there.
It looks before the probe goes, so the probe's answer comes to the new
socket (looking after it lost every answer, one socket after another).
A sender at rest does not look: only probes unanswered count.
make check moves a receiver (H3 of docs/design/011-pure-transport.md):
in namespaces of its own, the sender links to pure://receiver.test at
127.0.0.2; the receiver is killed, its store, key and memo copied, the
old address made unreachable, the name pointed at 127.0.0.3 and the
copy started there. Every order is applied once, and the moved receiver
makes no X25519.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
A listener lets a quiet sender's secret go with its active state, so a
node started again made one X25519 for every sender that came back (169
us on this Pi, most of a wake's CPU): asymmetric work in proportion to the
links, which design 011 counts as a reason to design again. The secret is
now remembered in a file beside the key (KEY.links, 0600, begun again for
another node's key): a place of 64 bytes a sender in the order allowed,
mapped, so a sender at rest takes no memory for it. A place is the first
28 bytes of the sender's key, a check (FNV-1a of the key and the secret)
and the secret, written secret first and key last; a place whose check
fails is made again. The UDP link checks see a receiver started again
make no X25519.
|
