Establish native micro runtime checkpoint

1 files changed, 165 insertions, 0 deletions

diff --git a/research/experiments/002-buffer-retention/README.md b/research/experiments/002-buffer-retention/README.md
new file mode 100644
index 0000000..d43940d
--- /dev/null
+++ b/research/experiments/002-buffer-retention/README.md
@@ -0,0 +1,165 @@
+# Experiment 002: buffer retention under backpressure
+
+This is a native C17 comparison apparatus, **not an implementation of the Pure
+runtime and not a Sonic, Node.js, or .NET benchmark**. It follows
+[research question 006](../../../docs/research/006-problem-landscape.md).
+
+Question: when a producer selects part of each record and a delayed consumer
+retains that selection, what costs remain after applying ordinary copy policies,
+bounded queues, pooling, a shared byte budget, and early field projection?
+
+## Work and observable output
+
+The CLI reads a sequence of experimental binary frames from stdin. Each frame
+has three little-endian uint32 fields: payload byte length, selection offset,
+and selection byte length, followed by the complete payload. The output is a
+little-endian uint32 selection length followed by the selected bytes. Input order
+is preserved. A payload is 1 byte to 8 MiB; a selection is nonempty and in bounds.
+The format has no magic, version negotiation, JSON syntax, or production claims.
+
+The Python generator materializes every byte, with deterministic seed, varying
+offsets, and source-size/selection-size distributions. It independently computes
+the expected output digest; tests additionally compare bytes against a Python
+reference decoder. Every measured run's complete output is checked. Input data is
+synthetic, not a production trace. Large temporary fixtures are reproducible from
+the saved plan, generator hash, and fixture manifest; they are not checked in.
+
+All variants read all payload bytes. `project` can discard bytes through a fixed
+64 KiB scratch area because this fixture's header supplies the selected range.
+That is an intentionally strong existing optimization when a format allows it.
+Its results do not establish that arbitrary JSON, compression, or a random-access
+document supports the same projection.
+
+## Variants and allocator
+
+| Variant | Decision |
+|---|---|
+| borrow | Keep the complete source buffer until output completes. |
+| copy | Copy the selected bytes to an independent buffer, then release the source. |
+| threshold2/4/8 | Copy when rounded source capacity / rounded selection capacity is at least 2/4/8. |
+| pressure | Copy a smaller selection when current live buffer charges exceed half the usable budget. No future lifetime information. |
+| project | Read the selection directly into its output buffer; read and discard the rest through scratch. |
+| copy_no_pool | Same copy policy without the reusable buffer cache. |
+
+`pressure` is a simple candidate online heuristic written in ordinary C. It is
+not described as a novel policy or Pure advantage. All thresholds were recorded
+before measurement; all results are reported, with no fitted or held-out claim.
+
+Blocks use power-of-two capacity classes, minimum 64 bytes. Pool-enabled variants
+reuse exactly matching classes. At most 16 idle blocks can be cached, consuming
+at most one quarter of the total byte budget. Idle blocks are evicted as needed
+before admitting another allocation. Pool-disabled runs use the same capacity
+rounding so that this comparison isolates reuse rather than size classes.
+This small explicit cache is an experimental baseline, not a best-in-class allocator.
+
+Each record has one owner at a time and there is one FIFO consumer. There is no
+fan-out, shared-owner reference counting, persistent result cache, or async device
+backend. Copy temporarily needs both source and destination; this overlap is
+charged. A budget too small to admit a record returns `budget_too_small` instead
+of waiting forever. Existing accepted output may already have been written.
+
+## Scheduling and budget
+
+The same single-threaded scheduler serves every policy. A queued output becomes
+eligible `hold_us` after enqueue. Consecutive output completions are separated by
+at least `service_us`. Ready output takes priority over another input record;
+otherwise the producer reads until a queue slot or byte credit is unavailable.
+Backpressure then services the oldest queued output, sleeping if needed.
+
+This uses actual reads, writes, allocations, copies, a monotonic clock, and
+`nanosleep`; it is not a simulated memory counter. The consumer delays are
+synthetic. A requested 20 us service interval is not a 20 us guarantee; kernel
+wakeup granularity and host scheduling affect the realized interval.
+
+The byte budget includes the engine, queue array, scratch, histogram, pool pointer
+array, block headers, active buffers, and cached idle buffers. It bounds requested
+storage owned by the apparatus, **not whole-process RSS**. It excludes allocator
+bookkeeping, libc/loader, executable pages, other stack frames, kernel I/O buffers,
+filesystem page cache, and the external Python verification process.
+
+## Metrics and measurement limits
+
+- `peak_reserved_bytes`: maximum fixed apparatus bytes plus active and cached
+  block charges. `peak_active_bytes` excludes cached idle blocks but includes the
+  fixed apparatus. Allocator bookkeeping is excluded from both.
+- `active_byte_seconds`: sum over active block leases of charged bytes times
+  elapsed lifetime. Includes transient input/output overlap and block headers;
+  excludes fixed apparatus storage and idle cache residency. Reused leases are
+  measured separately, without double counting a block's cached interval.
+- `copy_bytes`: only explicit selection `memcpy` bytes. Reads, writes, kernel,
+  allocator, and generator copies are outside this counter. Zero does not mean
+  no data movement in the process or system.
+- `latency_*`: starts immediately before reading a record header and ends after
+  its selected output is written. Includes admission waits after reading that
+  header. It excludes time a record waits outside the process before its header
+  is read. There is no independent open-loop arrival source, so these are not
+  service request P99 measurements. P99 is a histogram upper bound (up to 12.5%
+  bin rounding at normal durations), not an exact percentile.
+- `elapsed_ns`: file I/O, deliberate delays, scheduling, instrumentation and
+  buffer cleanup. Excludes build, process startup, fixtures and verification.
+  `cpu_ns` uses the process CPU clock; reports deliberate waiting separately.
+- `rss_peak_bytes`: approximate current-image `VmHWM` from Linux
+  `/proc/self/status`, read after the timed region. Zero with source `unavailable`
+  means no current-image implementation on another OS. `rusage_maxrss_bytes` is
+  retained separately (Linux KiB normalized to bytes; macOS already bytes).
+  Linux rusage survives exec and was contaminated by launcher history in the
+  first run; it cannot substitute for the child's current-image memory.
+  Neither measurement is added to requested buffer storage.
+
+Instrumentation is enabled equally in all variants, including block lifetime
+timestamps. Its overhead is not separately calibrated in this first experiment.
+Timing results must not be presented as uninstrumented application throughput.
+Output goes to a regular temporary file without `fsync`; this measures cached
+file I/O rather than durable storage. Each variant gets a warm-up run followed by
+five fresh-process measured runs in randomized variant order within each case.
+Cases run in fixed order. Linux runs pin to one available CPU; thermal and power
+conditions are uncontrolled. Medians and ranges are descriptive, not significance
+tests. The original plan predates the first measurement. Plan v2 adds an explicitly
+exploratory 64-slot sensitivity case after the first run exposed queue occupancy
+differences. Original data and source snapshots are preserved with an audit note;
+the v2 run is the current result. Neither plan is a production SLO.
+
+Lower peak RAM can permit a longer queue under the same byte cap and thereby
+increase observed in-process latency. Compare queue occupancy and latency as
+well as memory; equal byte/slot limits do not imply equal latency. No application
+SLO or success threshold has been invented for these synthetic inputs.
+
+## Reproduction and checks
+
+From the repository root, using Python 3 and a C compiler with ASan/UBSan:
+
+```sh
+python3 research/experiments/002-buffer-retention/run.py --check-only
+python3 research/experiments/002-buffer-retention/run.py --output /tmp/pure-retention-results
+```
+
+The result directory must not already exist. `CC` selects the compiler. The
+runner builds into a temporary directory, verifies normal and sanitized builds,
+and saves the plan, source hashes, build flags/hash/size, environment, fixture
+digests, checks, every raw sample and summaries. Build size includes debug info.
+`complete.json` appears only after all measurements and output hashes pass.
+LeakSanitizer needs an execution environment without ptrace tracing; its fatal
+diagnostics are treated as a failed verification, never suppressed.
+
+For direct use:
+
+```sh
+cc -std=c17 -O2 -Wall -Wextra -Wpedantic -Werror research/experiments/002-buffer-retention/benchmark.c -o /tmp/pure-retention
+/tmp/pure-retention --policy threshold --ratio 4 --budget 8388608 --hold-us 5000 < input.frames > selected.frames 2> metrics.json
+```
+
+Exit 0 indicates completion; exit 2 indicates an error; exit 3 indicates
+cancellation. SIGINT/SIGTERM cancel synchronous work and release outstanding
+buffers. The output may contain a completed prefix or a partially written final
+frame on I/O failure. This is not an atomic-output guarantee. Tests include exact
+output equivalence, empty input, malformed/truncated input, budget refusal,
+deterministic cancellation, real signal cancellation, and `/dev/full` where
+available. These synchronous tests do not verify async cancellation races.
+
+The RSS correction follows the documented preservation of
+[resource usage across exec](https://man7.org/linux/man-pages/man2/getrusage.2.html).
+Linux documents [VmHWM as approximate](https://man7.org/linux/man-pages/man5/proc_pid_status.5.html);
+requested storage counters remain the precise apparatus accounting used here.
+
+The first result and its interpretation are recorded in
+[report 007](../../../docs/research/007-buffer-retention-results.md).