docs(data): record DayOpen correctness and exact-column memory acceptance

2026-09-14 04:00:39 +08:00
parent 996b909608
commit 9a54156df9
2 changed files with 765 additions and 0 deletions
@@ -0,0 +1,117 @@
+# Exact Series Column Storage
+
+## Status
+
+Candidate not deployed. Twelve real long replays preserve their independent
+business baselines and reduce peak RSS by about 9.5%. Construction latency is
+mixed, including a reversed pair where the control is faster. This is accepted
+as evidence of a smaller working set, not as a proved general speedup or closure
+of the main performance objective. Original results are retained unchanged.
+
+The Source implementation remains d5b682c6d09704ff23d725a8dd8b155db3eb6967.
+Research/signal work remains paused. Current service remains the e81bf47/c98
+release; no Source, trading or other-user process was restarted or modified.
+The independent intraday-clock counterexample remains open.
+
+## Separate DayOpen Correction
+
+Code inspection found that PriceField::DayOpen selected the Open prefix sums,
+although direct history access returned day_open. For day_open values 10/12
+and open values 20/24, that path computes 22 instead of the expected 11.
+The correction adds its own day-open prefix and a regression checking both
+fields plus empty/insufficient windows. No price field is substituted.
+
+This correction was built and tested independently before the storage change:
+806 core unit/integration tests, 448 runner tests and 119 API tests passed.
+The resulting control runner is
+8859459f54389f12af1ab7d4e36802c01aff63fb10fbb679243ccdd54d013e2d.
+It also preserves the real rotation baseline. Both subsequent A/B variants
+include the fix, so corrected calculation semantics are not counted as speedup.
+
+## Storage Design
+
+SymbolPriceSeries previously allocated separate vectors for last/bid/ask,
+their prefix, timestamps, trading phases and three quote-volume fields, even
+when actual data repeated or exactly matched the existing close series.
+
+- ReferenceMatchedValues aliases the existing column only after every consumed
+  value matches by f64::to_bits. A mismatch materializes the exact preceding
+  values and continues as an owned vector. No missing/invalid price is replaced
+  by close; signed zero and NaN payload differences prevent sharing.
+- RepeatedValues retains the actual first value and logical length. It avoids
+  expanding equal values, including nonzero volumes and Some strings. The
+  first difference materializes the exact prior values. None is distinct from
+  an empty string; no value is inferred from the backtest frequency.
+- Intraday updates materialize only changed columns. Cloned views retain their
+  original values and immutable daily base. Last-price prefix sums use the same
+  accumulation order and actual values as before. History cutoffs are unchanged.
+
+There is no new dependency, unsafe code, strategy-specific branch, disk schema,
+source-data rewrite or account/result sharing. Construction and data validation
+remain in the existing paths. The overlay comment now accurately states that
+quote fields affect Last history while daily OHLC remains unchanged.
+
+The full candidate passes 813 core unit/integration tests (9 ignored), 448 runner
+tests (9 ignored) and 119 API tests (5 ignored). New tests cover exact bit
+identity, distinct zero/NaN values, repeated nonzero/string values, mutation
+isolation, unknown dates, full snapshot equality and history-date cutoffs.
+These tests do not prove the separately known same-day execution-clock issue.
+
+## Real A/B
+
+All cases execute 2021-08-23 through 2025-11-17 with their unchanged frozen
+strategy/runtime/bundle and 10,000,000 initial cash. This is not five complete
+execution years. Every run is a new process with private result artifacts and
+the same verified shared inputs: 9,257 files / 12,596,608,049 bytes. No original
+input changed and no Arrow/bin input was newly created. Hashing is outside the
+runner timer; no result is reused. Source/OS caches are not cold.
+
+| Case | Wall s | Data s | DataSet construction s | Engine s | RSS KiB |
+|---|---:|---:|---:|---:|---:|
+| Rotation control 1 | 23.879 | 5.213 | 1.901 | 6.612 | 7,137,676 |
+| Rotation candidate 1 | 24.126 | 4.818 | 1.589 | 6.674 | 6,463,660 |
+| Rotation control 2 | 18.180 | 7.272 | 2.824 | 9.509 | 7,138,628 |
+| Rotation candidate 2 | 15.078 | 5.543 | 2.094 | 8.171 | 6,454,624 |
+| Rotation candidate 3 | 13.125 | 5.103 | 1.853 | 6.649 | 6,457,728 |
+| Rotation control 3 | 12.725 | 4.955 | 1.713 | 6.583 | 7,140,772 |
+| Trend 40 control | 14.779 | 5.418 | 1.921 | 7.867 | 7,157,844 |
+| Trend 40 candidate | 15.012 | 5.168 | 1.851 | 8.011 | 6,470,768 |
+| Pullback 40 control | 14.011 | 5.172 | 1.893 | 7.298 | 7,167,408 |
+| Pullback 40 candidate | 13.877 | 5.117 | 1.837 | 7.219 | 6,478,492 |
+| Volume 80 control | 18.577 | 5.163 | 1.891 | 10.960 | 7,210,220 |
+| Volume 80 candidate | 18.476 | 5.051 | 1.844 | 10.989 | 6,527,236 |
+
+The final rotation pair deliberately ran candidate before control. Rotation
+RSS medians are 7,138,628 versus 6,457,728 KiB, about 665 MiB / 9.5% lower.
+Other strategy pairs save about 670-673 MiB. These are measured process peaks,
+not estimates obtained by adding cgroup limits or counting mmap as private RAM.
+
+Construction medians are 1.901 versus 1.853 seconds for rotation. The first pair
+has a larger reduction, but other samples and the reversed pair do not support
+a universal 16% construction or total-latency claim. Source validation waits and
+independent phase variation remain in the full evidence. A read-only host sample
+showed load near 49 and thermal readings 53/58/69 C; it does not prove the cause
+of timing variation. No host policy or another user's workload was changed.
+
+All six canonical sections and result-store SHA match the appropriate existing
+baselines: 21,393 / 29,776 / 31,862 / 51,300 fills. Result and request evidence,
+physical manifests and all 3,506 fact blocks were verified. No earlier failed
+or successful receipt was rewritten. Complete receipts remain on 177; only the
+compact verified summary is stored here to avoid duplicating input inventories.
+
+## Remaining Work
+
+Do not publish this as the main performance fix. Next, target the remaining
+daily snapshot/factor construction and direct typed-column reuse, avoiding
+new per-access branches or post-hoc compression passes. Cold-query acceptance,
+real minute-mode acceptance, HTTP publication and the intraday-clock issue are
+still outstanding. Signal lifecycle, financial PIT and UI factor conditions
+remain outside this completed storage experiment.
+
+- Engine candidate: 996b909608589fb1987f33c0cfb4c62099f69617.
+- Service source: 443ed421c2c9c854a01fab69ce58957690504570.
+- Candidate runner: 40bcf65c1977dbd93ab8bc80e3ff04d0db5e27b61fce1afdce99cf1b5e58eb43.
+- Candidate API: c54be3a8196c32051520c709f793bcb974d869467bb12700d846efaad8c2180e.
+- Evidence: /srv/fidc/canonical/run/research/series-column-storage-20260914.
+
+[Verified summary](evidence/series-column-storage-20260914/acceptance.json).