| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
| |
carry streams across module changes
|
| |
|
|
| |
must show (019)
|
| | |
|
| |
|
|
| |
research 119)
|
| |
|
|
| |
measure instead (018)
|
| | |
|
| | |
|
| |
|
|
| |
compare across machines (033)
|
| |
|
|
| |
bounded (018 stage 2)
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
Requests to an app that checks tokens go 2.92% slower than to an open
one on 64 keep-alive connections, 2.2% of it the head the token makes
longer (an open app sent the same head is as slow); hashing every
request on a new connection costs 0.91%. 031 measures the three in
turns.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
An app given a tokens file (tokens=FILE in --apps, --tokens FILE for the
one app of a host) answers a request only when it bears a token
(Authorization: Bearer) whose SHA-256 the file lists with a grant: read
for its views, queries and documents, write for all, input=NAME for
those inputs alone; 401 unauthorized (WWW-Authenticate: Bearer) without
one, 403 forbidden past its grant. A delivery sealed by a key --sender
names needs none. The digests are compared in constant time; a
connection keeps its last token's digest, so the same token is hashed
once. SHA-256 is the host's own (serve/sha256.c), no OpenSSL loaded. A
reading puts a changed tokens file in place with the app running on; a
file that is not one stops the app. pure-serve --make-token GRANT makes
a token and its line. load takes PURE_LOAD_TOKEN.
|
| |
|
|
|
|
|
|
|
| |
An app given a tokens file (tokens=FILE, --tokens FILE) takes requests
bearing a token whose SHA-256 the file lists: read for its views, queries
and documents, write for all, input=NAME for one input alone; 401
without one, 403 past its grant. A delivery another node sealed needs
none: its key told who sent it. A connection keeps its last token's
grant; a reading swaps the tokens without stopping the app.
|
| |
|
|
|
|
|
|
|
| |
The use after free in local_done and the owner's lock made after another
app's outlet used it were found reading the code, not by the check,
which passed with both before it was made to change senders (115, 017).
The usage now says changed lines change in place unless their store
moved; and 115 names the path no check takes: a module refused while
changing in place.
|
| |
|
|
|
|
|
|
|
|
| |
Changing an app in place keeps a million items' app answering but for
1.12 ms (a host restart: 150 ms; stopping and starting it again from its
store, as first built, 150.4 ms), 171.5 ms when its records change shape
(332 ms). Another app's p99 stays 0.97 times as it was while one changes
every 0.2 s; a host of one app is as it was (TPC-C +0.7%, instructions
at most +0.02%); each module change leaves 19 KB. 030 holds the windows
of both builds, the isolation, the pairs, the counts and the memory.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Stopping an app and starting it again reads its whole store again: a
million items took 142 ms, as long as restarting the host (030). A
running app whose line changed but not its store now goes quiet (what it
took applied, saved and answered, its batches back, its outlets closed,
its checkpoint finished), has its module replaced in memory
(pure_app_replace carries its states, and the sends in its outbox, by
name), its budgets set and its outlets made anew, and resumes; a module
its states do not go into stops it. An app whose store changed, one a
save of which is uncertain, and one that does not run still start from
their stores. A reading says changed for the first. A snapshot is
published again when the module changed though the states did not.
server_apps_lifecycle reads the changing apps' views (lists among them)
as they change, and moves an app's store; it passed 10 times alone, 5
under load and 6 with an ASan pure-serve.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
restart (030)
An app of a million items (a list grown by doubling) or of a thousand, in
a host of apps, its store in RAM: the longest gap between two answered
writes while it changes module on SIGHUP (stopped and started again from
its store, 71e319b), and while the host is restarted. A thousand items:
1.59 ms against 3.05 (median of 20 and 10). A million: 150.64 ms against
149.73 (10 and 5): both are the store read again (pure_app_start, 142
ms), so changing does not answer sooner than restarting.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A host of apps reads its --apps file again on SIGHUP, on a thread of its
own (pure-apps, the one that takes the signal): a file with a wrong line
changes nothing; the apps of lines gone stop and their routes go (404
no_app); those of changed lines, and those not running (they could not
start, or a save of theirs is uncertain), start again from their stores,
states carried by name; new ones take places of their own; a line out
says what was done. An app stops by taking nothing more (503 stopped),
applying, saving and answering what it took, waiting for its batches at
other apps, then closing its outlets (what they did not deliver stays in
the outbox) and its store. Its link streams find their inputs again as it
starts. Owners' locks are made with their places; counts are kept over
an app's runs; workers grow with the apps that run, a CPU each at most.
server_apps_lifecycle adds, removes and changes apps, and changes the
receivers (a local outlet's and a link's, their inputs at other numbers,
taken by name) and the senders while 600 sends each and writes go: every
send arrives once and in order, a write answered 200 is applied once and
one answered 503 stopped not at all, the app left alone fails no request;
an app whose store cannot be made starts at the next reading once it
can; kill -9 and the host starts as the file says. It passed 10 times
alone, 5 under load, 6 with an ASan pure-serve.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
A stream finds its input once, when it is made; an app that starts again
with another module may have the input at another number, of another
shape, or none. pure_link_receiver_rebind marks the streams to inputs of
a prefix stale: each finds its input again by name, with the fields it
was laid out with, before its next delivery and at its sender's next
packet, keeping what it holds; one none takes is refused as a new one
would be. pure_link_listener_rebind asks it of the listener's thread.
test-link's rebound case stops its app, starts it with the input at
another number, then of another shape taken by name: all 3,000 records
come once, in order (without the rebind, none after the stop).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
stage 3)
Groundwork for apps that start, stop and change on a running host, with
no change in what a host does:
- an app's module is on the heap, and the module its requests are
resolved with is published (live, NULL while it does not run); a
module it ran is kept while the host runs;
- a request, a view, a link's binding and a local outlet carry the module
they were resolved with, and an owner takes none resolved with another
(503 stopped); a local outlet finds its input again when its receiver
runs another module, and holds its sends while it runs none;
- the published snapshot is the owner's, outliving an executor;
- apps take places, made when the host starts (--app-slots, 256), never
given to another name, so threads find apps by name as others start.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
(017, stage 3)
The --apps file is the host's shape: on SIGHUP the host reads it again
and starts the apps of new lines, stops those of lines gone, starts
again those whose line changed (states carried by name from the store)
and those stopped or uncertain. A request carries the module it was
resolved with, and is not applied under another (503 stopped); link
streams and local outlets look their input up again. Route
authentication (3b) is designed apart.
|
| |
|
|
|
|
| |
The same events make byte-identical stores on the x86_64 WSL2 (4 KiB
pages) as on the Pi and the Mac, and a store made on either of the Pi
and the WSL2 opens on the other and goes on the same.
|
| |
|
|
|
|
|
|
|
| |
The same modules and inputs give the same outputs, failures and store
bytes on the Pi, an x86_64 WSL2 and an M3 Mac; other compilers found an
argument read before it was set, and the core and store build on macOS
again. WSL2's wall clock steps back about 2 s every 32 s. The Pi froze
twice under a measuring tool that held its output: it now streams it,
and experiments run under research/guard.py.
|
| |
|
|
|
|
|
|
|
|
| |
Every pure-run of aup's check (1,053 runs of 667 modules) gives the same
output, errors and exit on the Pi, the WSL2 (x86_64, 4 KiB pages, gcc 13)
and the Mac (Apple clang 21); the same events make byte-identical stores
on the Pi and the Mac, and either opens the other's and goes on the
same. Pure's checks pass one at a time in WSL (each_check.py); its wall
clock is set back about 2 s every 32 s (clock.py). stores.py streams
pure-run's output, whose size grows with the square of the events.
|
| |
|
|
|
|
|
|
|
|
|
| |
The Pi has 8 GiB, zram swap and no OOM daemon, and the user's systemd
holds no memory controller: a run that outgrows its memory has every
program's code evicted before the kernel kills anything, and the
machine stops answering. It was reset twice so (a run held 13 GiB of
output it should have streamed). guard.py runs a command in a session
of its own and kills the session when it holds more than --most MiB
(3072) or the machine has less than --floor MiB available (1024, a
tmpfs filled counts), and stops what it leaves running.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Apple clang 21 (an M3) and clang 19 refused what gcc let pass (research
029):
- PURE_FIELD_REST was an enum of UINT32_MAX, past an int (C17): a macro.
- a store's publish copied an entry rebuild left unset when the record
was not found (the copy was never used): copied when it was made.
- the log's image writer kept a count only Linux's sync_file_range uses.
- ",{" + (row == from) read as adding an int to a string: said plainly.
- the tests: mkdtemp, hidden by _POSIX_C_SOURCE on macOS, needs
_DARWIN_C_SOURCE; a string split so that \x80 and a stay apart is
bracketed; a count left over and a malloc measure only glibc has.
The core's and the store's tests pass on the Mac.
|
| |
|
|
|
|
|
|
|
| |
server_timers compares the times an app saw with the wall clock read
around them. WSL2's clock is set back about 2 s every 32 s (research
029): a window holding such a step says nothing of the times seen, and
failed the check there 4 times in 20. It is taken again (up to three
times) when the wall clock and the monotonic one moved more than 50 ms
apart over it; the report counts the windows taken again.
|
| |
|
|
|
|
|
| |
Ubuntu's gcc defines _FORTIFY_SOURCE, which marks write's result as one
to use, and a (void) cast does not quiet it: the prototype did not
build there (research 029). Debian's does not define it; the Pi gives
the same error with -D_FORTIFY_SOURCE=3.
|
| |
|
|
|
|
|
|
| |
pure_list_get passed leaf_of(list, at, &within) and within to cell_at
in one call: C leaves the order of a call's arguments open, so within
could be read before leaf_of set it. gcc 14 on AArch64 called leaf_of
first; gcc 13 on x86_64 says within is used uninitialized (found
building Pure there, research 029).
|
| |
|
|
|
|
|
| |
The loop alone slowed four links to half: woken by every record queued,
it chased the owner a record at a time. Pushing once a turn made links
1.6 times as fast, with threads or the loop alike; the loop's own cost
is within the noise. Twenty outputs now add two threads, not 21.
|
| |
|
|
|
|
|
|
|
| |
Five rounds of 1,000 bursts by one wire, four wires, one link and four
links: a thread each (966b82f), links on the loop (94e7bb8), wires on
it too (5f5603c). Medians: one wire 774k, 782k, 779k sends/s; four
wires 629k, 625k, 615k (the same CPU a send); one link 581k to 948k;
four links 485k to 811k. A host whose app sends to ten links and ten
wires has 24 threads before and 5 after (3 with no output).
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
A wire is now a sender of link/sending.h: its request goes on a
non-blocking socket as far as the socket takes it (connect, TLS
handshake, write, answer read), and its round waits for the socket or
its deadline (10 s with nothing moved). The answer is read as it comes:
the head whole, then a sized body counted down, a chunked one (few and
small) parsed again as it grows, or a body to the close. The crypto
table (version 6) takes the TLS handshake a step at a time, and reads
and writes say what they wait for. The owner pushes a wire once a turn,
as it does a link. A host whose app sends to ten links and ten wires
has 5 threads where it had 24 (none: 3).
|
| |
|
|
|
|
|
|
| |
Five rounds of 1,000 bursts by one link, four links and one wire, four
builds in turns: a thread a link (966b82f), those threads with push
(threads-push.patch), the loop (99b2219), the loop with push (94e7bb8).
Medians: one link 597k, 923k, 525k, 897k sends/s; four links 506k,
799k, 268k, 800k; the wire as it was, 789k to 794k.
|
| |
|
|
|
|
|
|
|
| |
link/sending.c is the process's sending loop: a sender joins it with its
round and the socket it waits on, and is rounded when its socket is
ready, when pushed, and when its time comes. Links are its first
senders; nothing they do changed. Events of a sender that left while the
loop waited are let go (the epoll is level-triggered: the others' come
again).
|
| |
|
|
|
|
|
|
|
|
|
|
| |
one (017, 2b-1)
The sending loop was woken by every record queued while it slept: the
owner queues one by one, the loop caught up and slept between them, and
every link took a round on every wake. Four links sent 64k packets of
12 records where threads sent 40k of 20, and ran at 257k sends/s
against 581k. The owner now queues a turn's sends and pushes each link
once; the loop rounds a link only when its socket has answers, it was
pushed, or its time came.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A link sent on a thread of its own, polling its socket and a wake. Now
the process has one sending loop (pure-send), made with the first link
started and stopped with the last closed: an epoll over the links'
sockets and one wake, sleeping until the soonest link's next time. Each
round does for every link what its thread did: answers taken when its
socket had them, its times kept, the receiver looked for again when
probes go unanswered (a new socket joins the epoll), packets sent, and
the owner told of what was stored. A queue wakes the loop only while it
sleeps, once. Events name links by number, so one closed since its
event came is not touched; answers are received into the loop's one
buffer.
A name looked up again (after three probes unanswered) waits where it
is, and every link waits with it; it was so for its own link before.
Every check passes as it did.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Each link and each HTTP wire sends on a thread of its own: a host with
twenty outputs runs twenty-one threads more than one without. 2b moves
them into one sending loop: an epoll over the links' UDP sockets and
the wires' TCP sockets with one wake, waiting until the soonest link
timer or wire pause. Links (2b-1) do in the loop what their threads
did each round; wires (2b-2) become a non-blocking HTTP/1.1 client
state machine, with non-blocking steps added to the crypto table for
TLS. A name looked up again still waits where it is, and says so.
Before building: every link, wire, answer and seal check green; a host
of ten links and ten wires two threads above one without; and link and
wire throughput, alternated before and after, no more than 5% lower at
the median (four links together: 10%).
|
| |
|
|
|
|
|
|
| |
Stage 2 is built and measured: what was learned (the receiving side
needed nothing new, an output goes one way only) and the numbers. The
checkpoint adds sending to the many-apps row, and keeps 2b (one loop
for the send threads) and stage 3 (apps added and replaced while the
host runs) among what is left.
|
| |
|
|
|
|
|
|
|
|
| |
Two apps of one host, src sending one message for each of 1,000 items
in a burst event, 2,000 bursts a run, five rounds with the ways in
turns: the local outlet stores 1,141,642 sends a second at 1.17 us of
the host's CPU each, a loopback pure:// link to the host's own port
578,879 at 2.93 us (1.97 times the sends, 0.40 times the CPU). 017's
second stage asked for at least the link's rate at half its CPU or
less: it goes in. Every run checks dst's sum: each send came once.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
send=OUTPUT=APP/INPUT in --apps wires an output to another app of the
same host through a local outlet: no socket, seal or packet. The
sender's turn queues its saved sends there, under the window links
and wires have, and hands them, one batch at a time, to the receiving
app's queue laid out as a link delivers records (first number, count,
source APP.OUTPUT, the records as stored, and the sender's fields when
the input takes them by name, which pure_module_takes checks at start).
The receiver applies them after its last number heard; its save over,
it tells the outlet what it heard and the sender's next turn trims its
outbox so far. What it did not store is queued again from the outbox,
after 100 ms when it had no room. An output now goes one way only: two
outlets would trim each other's sends.
server_apps_links: an app sends each of 400 messages to an app of
another host over a link and to three apps of its own host (one of
the same shape, one taking the fields by name, one with a memory
budget that fills), with a kill -9 of the sending host between two
requests; all arrive once and in order, and the full app's sends wait
in the sender's outbox until it lets go of what it holds.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
An app of --apps takes send=OUTPUT=URL, seal=OUTPUT=KEY and
reply=OUTPUT=INPUT, as --send, --seal and --reply give the one app of a
host; its sends' source is APP.OUTPUT. An app wired wrong stops alone,
its wires closed. The host's --key, --sender, --senders, --sealed-only
and --link-port now go with --apps: the one listener resolves a
stream's input name APP/INPUT to that app's input (a stopped app or an
unknown name is no input, REFUSED 1), its deliveries queued at that
app's owner. 012 is unchanged.
server_apps_links: an app of one host sends 300 messages, sealed, over
a link to an app of another host, which takes them all once and in
order.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
show (017)
Stage 2: each app of --apps takes its wiring (send=, seal=, reply=), its
sends' source being APP.OUTPUT; one --link-port takes every app's
inputs, a stream's input name APP/INPUT (012 unchanged); and send=
OUTPUT=APP/INPUT hands an app's saved sends to another app of the same
host with no socket, seal or packet: an outlet with the window links
have, a batch of records as a link delivers them put in the receiving
app's queue, applied after its last number heard, and the number heard
after its save trimming the sender's outbox. Merging the send threads
into one loop is split off as 2b.
Before building: it goes in if the inner outlet stores at least as many
sends a second as a loopback pure:// link between the same two apps,
at half the host's CPU a stored send or less; checked by a host sending
to an app of its own and to one of another host, exactly once and in
order across a kill -9, and held back by a full receiver.
|
| |
|
|
|
|
|
|
|
|
| |
017's first stage is built and measured: the shape now says what was
built where it differs from the design (work counted as written,
nothing counted without a budget, every app of --apps run by workers,
a busy app kept on its worker, a home thread for each app's
connections), and what was learned on the way. The checkpoint lists
many apps on one host among what Pure shows, and stages 2 and 3 among
what is left.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
017's first stage met its four criteria on the Pi (RAM stores, five
rounds, the order of the ways turned): app B's 99th percentile while
app A is flooded by 64 connections is 168 us, half of two processes'
336 us; the two apps busy together write 146.1k a second against
146.2k, with the same 1.4 ms tail; an app costs a host 120 KiB of its
own memory (a process: 1,311 KiB PSS) and 100 apps still run on three
threads; a host of one app keeps its speed (TPC-C +1.2% at the median
of six pairs, two of them interfered and kept; the bank's queries
within 2.6%) and its instructions (+0.94% at most).
On the way: apps hopped between workers each turn (fixed in db4ee9c),
connections of one app spread over all connection threads made the
host a third more expensive a request and its tail a kernel tick
(fixed by home threads, d24881d), and the work budget's counting cost
every table row 26 instructions (fixed in be297ea). The earlier runs
are kept apart (db4ee9c's isolation, d24881d's pairs and counts, the
interrupted first try).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
The work budget added about 26 instructions to every row of a table a
host shows (bank, a table of 10,000 rows: +3.0% instructions an ask),
which 017's one-app criterion allows at 1%. A sample put most of it in
show_given asking every cell why it failed, and the rest in evaluate
and the row lookup writing that reason and the count for every cell.
Now nothing is counted without a budget (PureApp.work stays), the
reason is written only when a view, a cell or a row has no value, and
pure-serve asks it only of a cell without one. The table: +0.94%; the
other bank queries within 0.4%. The tests that read the count set a
budget that is never reached.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With several connection threads, a host of apps had each app's
connections spread over all of them: every thread's round handed each
owner a share, and every turn handed answers back to every thread. The
apps took a third more CPU a request than as processes, with five times
the voluntary switches, and when two apps were busy each waited a
kernel tick at its 99th percentile (4.1 ms, against 1.5 ms as two
processes).
Each app now has a home thread (its place in --apps, modulo the
threads). A connection whose first request names an app at home
elsewhere moves there once, its head read and no ticket taken yet: the
home thread adopts its descriptor and the bytes read, and reads the
request again. Each app's requests then meet its owner from one
thread, as from a process of its own. Two busy apps: 99th percentile
1.4 ms, the same 145k writes a second.
server_apps restarts its apps on three connection threads, asks each
app from fresh connections, and checks that some moved and all were
answered right.
|
| |
|
|
|
|
| |
load takes a ninth argument, PREFIX, put before every path: /NAME for
an app of a host of apps (pure-serve --apps, 017). Without it, as
before.
|
