| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The app model, the tree store and the HTTP host lived in
prototypes/connections among the earlier document runtime, the store was
known only by its format's name, and Vega compiled six Pure sources by path.
Now, as the README's principle has it:
core/ runtime, flow and the app model (module*.c), with what the core
asks of storage (state.h, store.h)
store/ the tree store (PURETRE3, PUREREC1)
serve/ pure-serve: HTTP host, outbound delivery, JSON in place
cli/ pure-run, pure-edit-module
A top-level make builds libpure.a (app model and store, for hosts to link)
and the hosts; make check runs the store's and the app model's tests. The
earlier runtime still builds from prototypes/connections against core/ and
store/ (check, check-tree and check-core pass). Design 009 records the rule:
parts by role behind libraries; git repositories split by coupling, so the
store stays here while its contract moves with the core, and Vega stays apart.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A tree store commit wrote its bytes over the written zeros of the tail
and its head into a slot at the start of the file: two ranges, and on
the Pi's SD card a sync of two ranges costs about twice one (2.4 ms
against 1.2 ms). A commit now writes a 128-byte record (its head) at
the published end and its bytes right after it, then syncs once. Every
16 commits the head also goes into the older slot. The newest commit
is found from the newer slot's head, or from the newest head this
process has seen in the file while its record is still there, along
the records whose bytes are whole; a torn record or bytes end the walk
at the commit before, which the next writer overwrites. Slots of this
format are PURETRE3; files of the earlier one are read and continued.
Store commits went from 220-233/s to 371-396/s (p50 2.5 to 1.3 ms),
and the scoreboard's durable posts from 13-16k/s to 21-26k/s with 64
connections and from 2.2-2.4k/s to 2.9-3.9k/s with 8, measured
alternately with the previous build. The torn head check now tears a
slot (the walk goes on from the other) and the newest record (a new
process sees the commit before); a new check walks 42 commits from the
slots, in a fresh process too, and stops at a damaged record.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Right after a save, pure_app_snapshot makes a read-only snapshot of the
saved states: numbers are copied, text states are frozen drafts that
share the store mapping (pure_draft_freeze, no copy), and lists are
copies of their item pointers that share the items. An unchanged list
shares the last snapshot's copy. pure_app_read binds a reader app on
another thread to a snapshot so the existing view functions read it.
A shared item never changes again: a list stamps each item with its
share count, copies a shared item it changes, and retires shared items
it lets go instead of freeing them. Snapshots form a chain that goes
oldest first, and items retired while a snapshot was the newest go
with it. References are atomic and may be released on any thread.
test-module checks that each snapshot equals the app, that kept
snapshots stay as made through changes, removals, a module switch, the
app stopping and the store closing, that a failed save's snapshot
equals the one before, and that two threads read the newest snapshots
while the app goes on (also under ASan, UBSan and LeakSanitizer).
|
| |
|
|
|
|
|
| |
A read's lent extents hold the store mapping until its answer is sent;
with an owner thread running the app and connection threads answering,
the last reference may drop on a thread other than the one that took it.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A commit used to sync its nodes, then write the head and sync again: the
order of writes was forced by syncs. Now the head carries where the
commit's bytes start and their digest, and one fdatasync makes the bytes
and the head durable together. A crash during it can leave a head whose
bytes did not all reach the device; its digest then fails and the older
slot stands, which is safe because a writer only builds on a durable head.
Commits overwrite written zeros kept after the published end, so the sync
changes neither the size nor the block map and needs no journal commit.
On the SD card this is 2.4 ms per commit against 9.1 ms for two syncs;
the store commits 289-304 times a second instead of 94-109, and pure-serve
answers 2,012-2,611 durable PUTs a second against 583 for crash-safe axum.
A process remembers the heads it published or checked, so it reads a
commit's bytes at most once and never its own. Its readers pass over the
head it is still making durable and so never answer with a commit a crash
could take back. Head slots get a block each and the format becomes
PURETRE2; a handle's refresh reads the slots through its mapping.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
A read of a stored document now answers with the store mapping's extents
through writev, without copying the document. pure_draft_mapping takes a
reference to the mapping a draft opened through a handle reads from; the
answer keeps it until the last byte is sent, so the version read goes out
intact even if the document changes or a compaction replaces the file
meanwhile. Replies the mapping does not hold are copied as before.
pure-serve's output is now a list of extents (the head first). One thread
serves 2,761 1 MiB reads per second (axum: 1,380) and 128k small ones.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Threads that commit at the same time now share one commit: they queue,
the first becomes the leader and publishes every queued request of the
same store with one head, and the others wait for its result. Each
request stays all or nothing on its own: one with a stale draft is left
out, and so is one whose key an earlier request of the group changes.
When several threads have been committing together, a leader waits up to
2 ms for as many to join as groups held lately; a single thread never
waits.
A test pauses a publication and checks that five commits make exactly two
publications. Under eight writers on three pure-serve shards, 1,234
bursts went out in 421 publications. pure_store_group_counts reports the
groups, and pure-serve prints them.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A burst now looks at the store once (pure_store_pin) instead of once per
request; another shard's commit ends before its answer is sent, so a
request made after that answer still reads it. With this alone one thread
serves 125k small keep-alive reads per second, as many as axum on three
cores, with a shorter p99.
--threads N runs a shard per thread: its own SO_REUSEPORT listener, epoll
set, runtime domain, app and store handle. Writer sessions of the tree
store now take turns inside a process with a mutex (the file lock belongs
to the process and does not keep threads apart), and revisions settle
concurrent edits of a key. A module with states runs on one thread. Three
shards serve 151k reads per second; they still commit one by one, so
durable writes and 1 MiB reads are better on one thread for now, which
stays the default. Note 075 records the numbers.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Opening a draft by path opened the directory and the file, mapped the
whole file, read the head, and undid all of it on close: nine system
calls for every GET. pure_store_open keeps the store open for one owner
thread; drafts opened through it share a reference-counted mapping of the
newest version, and opening one only checks whether the file changed (its
identity and its head: two system calls). A commit of its drafts maps the
new version once and moves them all onto it. A compaction by another
writer is noticed and followed, while drafts of the old file keep reading
it and relocate their extents when they commit.
pure_store_data gives modules the same through a handle.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
pure_draft_commit_all publishes drafts of one store with a single head:
every key must still be at its draft's base revision, all new extents and
catalog records are written and synced once, and either every document
becomes visible or none does. A single draft commit is the one-draft case.
pure_state_open stores runtime-owned values (app states) under '.' plus a
key. Document keys never start with '.', listings skip these values, and
document operations refuse their names, so an app's state cannot collide
with a document a client names.
Tests kill a process at every commit stage of a three-draft batch and see
all or none, inject each fault, stop a batch on one stale draft, and
commit fifty keys in no order at once.
|
| |
|
|
|
|
|
|
|
|
| |
Runtime-owned values such as an app's text need the same extent tree as
stored documents without a store file. pure_draft_new creates such a draft;
it edits, lends versions and reclaims like any draft but refuses to commit.
pure_draft_restore returns a held draft to an earlier version, so a failed
event can abandon its edits. A test runs 4,000 random edits with a fifth
abandoned against a flat oracle under ASan/UBSan.
|
| |
|
|
|
|
|
|
|
| |
A module event may read a document and then edit it; the value it read must
not change. pure_draft_version captures an immutable version and
pure_draft_read_version lends its extents; pure_draft_hold suspends arena
reclamation while a transaction still reads earlier versions. A test reads
the original 200 KB after 400 held edits and checks reclamation on release.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
state.h gives any app a runtime-owned working version of a stored value on
the tree store. A draft shares unchanged extents with the stored version,
edits in memory by path copying, lends ordered extents without copying, and
commits only its new nodes when the key is still at its base revision. A
failed commit changes neither the draft nor the store; a draft whose file was
replaced by compaction copies its stored extents on commit. Unreachable
arena nodes are reclaimed once they outweigh max(reachable, 32 KiB).
Typing into the middle of a 1 MiB document and saving each character peaked
at about 1,660 KiB RSS with a draft against about 4,480 KiB for an app copy
compared with storage on save; saves were only 1-2 ms faster. This corrects
073, which had attributed the TUI's remaining 8 ms to that comparison.
Drafts are not yet charged to the runtime budget or visible to .pure
definitions.
|
|
|
store-tree.c keeps every document in one append-only file as a persistent
B-tree of byte extents, with the key catalog in the same structure. A splice
writes only the changed leaf and path; unchanged extents are shared between
versions, readers take no lock, and two alternating head slots publish a
commit after its nodes are durable. Compaction rewrites reachable nodes.
On the recorded ext4 host a durable 1-byte middle insert into a 1 MiB
document took 8.58 ms against 78.51 ms for the default store, and the real
TUI's Ctrl+S after typing took 16.60 ms against 84.59 ms. check-tree runs the
store's crash/oracle tests and the unchanged runtime, splice, async and
real-PTY editor checks on it. It stays opt-in: linked code is 12.7 KB larger
and contiguous multi-leaf reads copy.
|
