fidc-backtest-engine/docs/series-column-storage-performance-20260914.md

# Exact Series Column Storage

## Status

The subsequent business-main merge includes the separately published 81acc54
clock/callback fixes and 5e11f3d manual-replay foundation. The combined version
passed 857 core, 448 runner and 119 API tests, six long reference replays and
three additional strategy replays;
see `/Users/boris/WorkSpace/fidc-backtest-service/docs/arrow-factor-scratch-rejection-20260914.md`.
The scratch candidate from that experiment was removed. Series storage is
still not deployed; existing measurements below retain their original versions.

Candidate not deployed. Twelve real long replays preserve their independent
business baselines and reduce peak RSS by about 9.5%. Construction latency is
mixed, including a reversed pair where the control is faster. This is accepted
as evidence of a smaller working set, not as a proved general speedup or closure
of the main performance objective. Original results are retained unchanged.

The Source implementation remains d5b682c6d09704ff23d725a8dd8b155db3eb6967.
Research/signal work remains paused. The initial experiments ran while e81bf47/c98
was published; later business work published e81bf47/81acc54. This performance
task did not restart Source, trading or another user's process. The original
clock counterexample was resolved by that business work; complete manual-replay
integration remains open and is not proved by these performance tests.

## Separate DayOpen Correction

Code inspection found that PriceField::DayOpen selected the Open prefix sums,
although direct history access returned day_open. For day_open values 10/12
and open values 20/24, that path computes 22 instead of the expected 11.
The correction adds its own day-open prefix and a regression checking both
fields plus empty/insufficient windows. No price field is substituted.

This correction was built and tested independently before the storage change:
806 core unit/integration tests, 448 runner tests and 119 API tests passed.
The resulting control runner is
8859459f54389f12af1ab7d4e36802c01aff63fb10fbb679243ccdd54d013e2d.
It also preserves the real rotation baseline. Both subsequent A/B variants
include the fix, so corrected calculation semantics are not counted as speedup.

## Storage Design

SymbolPriceSeries previously allocated separate vectors for last/bid/ask,
their prefix, timestamps, trading phases and three quote-volume fields, even
when actual data repeated or exactly matched the existing close series.

- ReferenceMatchedValues aliases the existing column only after every consumed
  value matches by f64::to_bits. A mismatch materializes the exact preceding
  values and continues as an owned vector. No missing/invalid price is replaced
  by close; signed zero and NaN payload differences prevent sharing.
- RepeatedValues retains the actual first value and logical length. It avoids
  expanding equal values, including nonzero volumes and Some strings. The
  first difference materializes the exact prior values. None is distinct from
  an empty string; no value is inferred from the backtest frequency.
- Intraday updates materialize only changed columns. Cloned views retain their
  original values and immutable daily base. Last-price prefix sums use the same
  accumulation order and actual values as before. History cutoffs are unchanged.

There is no new dependency, unsafe code, strategy-specific branch, disk schema,
source-data rewrite or account/result sharing. Construction and data validation
remain in the existing paths. The overlay comment now accurately states that
quote fields affect Last history while daily OHLC remains unchanged.

The full candidate passes 813 core unit/integration tests (9 ignored), 448 runner
tests (9 ignored) and 119 API tests (5 ignored). New tests cover exact bit
identity, distinct zero/NaN values, repeated nonzero/string values, mutation
isolation, unknown dates, full snapshot equality and history-date cutoffs.
These tests do not prove the separately known same-day execution-clock issue.

## Real A/B

All cases execute 2021-08-23 through 2025-11-17 with their unchanged frozen
strategy/runtime/bundle and 10,000,000 initial cash. This is not five complete
execution years. Every run is a new process with private result artifacts and
the same verified shared inputs: 9,257 files / 12,596,608,049 bytes. No original
input changed and no Arrow/bin input was newly created. Hashing is outside the
runner timer; no result is reused. Source/OS caches are not cold.

| Case | Wall s | Data s | DataSet construction s | Engine s | RSS KiB |
|---|---:|---:|---:|---:|---:|
| Rotation control 1 | 23.879 | 5.213 | 1.901 | 6.612 | 7,137,676 |
| Rotation candidate 1 | 24.126 | 4.818 | 1.589 | 6.674 | 6,463,660 |
| Rotation control 2 | 18.180 | 7.272 | 2.824 | 9.509 | 7,138,628 |
| Rotation candidate 2 | 15.078 | 5.543 | 2.094 | 8.171 | 6,454,624 |
| Rotation candidate 3 | 13.125 | 5.103 | 1.853 | 6.649 | 6,457,728 |
| Rotation control 3 | 12.725 | 4.955 | 1.713 | 6.583 | 7,140,772 |
| Trend 40 control | 14.779 | 5.418 | 1.921 | 7.867 | 7,157,844 |
| Trend 40 candidate | 15.012 | 5.168 | 1.851 | 8.011 | 6,470,768 |
| Pullback 40 control | 14.011 | 5.172 | 1.893 | 7.298 | 7,167,408 |
| Pullback 40 candidate | 13.877 | 5.117 | 1.837 | 7.219 | 6,478,492 |
| Volume 80 control | 18.577 | 5.163 | 1.891 | 10.960 | 7,210,220 |
| Volume 80 candidate | 18.476 | 5.051 | 1.844 | 10.989 | 6,527,236 |

The final rotation pair deliberately ran candidate before control. Rotation
RSS medians are 7,138,628 versus 6,457,728 KiB, about 665 MiB / 9.5% lower.
Other strategy pairs save about 670-673 MiB. These are measured process peaks,
not estimates obtained by adding cgroup limits or counting mmap as private RAM.

Construction medians are 1.901 versus 1.853 seconds for rotation. The first pair
has a larger reduction, but other samples and the reversed pair do not support
a universal 16% construction or total-latency claim. Source validation waits and
independent phase variation remain in the full evidence. A read-only host sample
showed load near 49 and thermal readings 53/58/69 C; it does not prove the cause
of timing variation. No host policy or another user's workload was changed.

All six canonical sections and result-store SHA match the appropriate existing
baselines: 21,393 / 29,776 / 31,862 / 51,300 fills. Result and request evidence,
physical manifests and all 3,506 fact blocks were verified. No earlier failed
or successful receipt was rewritten. Complete receipts remain on 177; only the
compact verified summary is stored here to avoid duplicating input inventories.

## Remaining Work

Do not publish this as the main performance fix. Next, target the remaining
daily snapshot/factor construction and direct typed-column reuse, avoiding
new per-access branches or post-hoc compression passes. Cold-query acceptance,
real minute-mode acceptance and HTTP publication remain outstanding for this
storage candidate. The original clock issue was fixed by subsequent business
work, not this experiment. Signal lifecycle, financial PIT and UI factor conditions
remain outside this completed storage experiment.

- Engine candidate: 996b909608589fb1987f33c0cfb4c62099f69617.
- Service source: 443ed421c2c9c854a01fab69ce58957690504570.
- Candidate runner: 40bcf65c1977dbd93ab8bc80e3ff04d0db5e27b61fce1afdce99cf1b5e58eb43.
- Candidate API: c54be3a8196c32051520c709f793bcb974d869467bb12700d846efaad8c2180e.
- Evidence: /srv/fidc/canonical/run/research/series-column-storage-20260914.

[Verified summary](evidence/series-column-storage-20260914/acceptance.json).