arXiv:2609.32041 · cs.CV · 25 September 2026
Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligence
Survey with a formal framework, a benchmark audit and an evaluation harness. Document AI reads, then forgets; this survey argues the fix is not more scale, context or retrieval, but persistent, provenance-linked state, and specifies how to build and measure it. Every figure below is interactive.
§ I Introduction
The statelessness bottleneck
Document systems are modeled as single-shot functions, ŷ = argmax P(y | D; θ): frozen parameters, no session index. Whatever one document taught is gone when the next one arrives.
Document AI reads, then forgets
Systems extract fields, reason over tables and ground answers to page regions, yet process each document as if it were the first. An amendment read the next day starts from scratch: no schema retained, no contradiction detected.
It is a design choice, not a scale failure
Current systems are stateless functions, y = argmax P(y | D; θ), with frozen parameters and no session index. The paper calls this the statelessness bottleneck.
Scaling, context and retrieval do not remove it
Storage provides persistence and retrieval provides access, but neither consolidates observations into knowledge that improves how the next document is processed.
So the paper specifies persistent, evidence-grounded state
It formalizes the state, its lifecycle and invariants, bounds the risk of remembering wrong things, audits ten benchmarks, and proposes a longitudinal harness with five counterfactual metrics.
D = (V, T, B, G, H, M)
- V
- Visual tensors
- T
- Text tokens
- B
- Bounding boxes
- G
- Structural relation graphs
- H
- Hierarchical parse trees
- M
- Metadata
All six properties are required jointly; no adjacent paradigm has them all.
- Multimodal evidence
- Document structure
- Persistence
- Temporal validity
- Provenance
- Governance
§ II Capability evolution
Where the field stands: an empty top row
Fig. 2 places Document AI lineages by how much they persist. Cross-modal models stop at the page with no persistence; retrieval reaches an external store; text-only agent memory reaches episodic state. Nothing reaches semantic or governed state.
Target, spanning the Governed and Semantic bands: 𝒞*
- Multimodal evidence
- Document structure
- Persistence
- Temporal validity
- Provenance
- Governance
-
GovernedEmpty: no surveyed system reaches this band
-
SemanticEmpty: no surveyed system reaches this band
-
Episodic
-
Text-only agent memory , scope: Conversation / session
- MemGPT
- LoCoMo
- SYNAPSE
- GAAMA
-
-
External store
-
Retrieval systems , scope: Collection
- ColPali / ViDoRe
- VisRAG
- GraphRAG
- LightRAG
-
-
No persistence
-
Cross-modal & layout models, single-page benchmarks , scope: Page
- DocVQA
- ChartQA
- FUNSD
- LayoutLM
- Donut
- EAML
- SelfDoc
- UDoc
- VLCDoC
- GlobalDoc
-
Multi-page benchmarks , scope: Multi-page document
- MP-DocVQA
- DUDE
- LongDocURL
- M-LongDoc
-
§ II-B Representation lineage
A decade of representations, no native state
Tables I and II: thirteen representation models and four adjacent paradigms, against the properties the target needs.
| Method | Year | Modality | Pretraining | Layout | Visual | OCR-free | Native state |
|---|---|---|---|---|---|---|---|
| BERT | 2019 | Text | MLM | – | – | – | – |
| LayoutLMv1–v3 | 2020–22 | Text+Layout+Vision | MLM/MIM | ✓ | ✓ | – | – |
| EAML My work: EAML: Ensemble Self-Attention-Based Mutual Learning Network for Document Image Classification | 2021 | Image+Text | Mutual learning | – | ✓ | – | – |
| SelfDoc | 2021 | Cross-modal | Self-supervised | ✓ | ✓ | – | – |
| UDoc | 2021 | Cross-modal | Unified alignment | ✓ | ✓ | – | – |
| DiT | 2022 | Vision | MIM | Implicit | ✓ | ✓ | – |
| Bi-VLDoc | 2022 | Cross-modal | Bidirectional VL | ✓ | ✓ | – | – |
| Donut | 2022 | Vision→Text | Seq2seq | Implicit | ✓ | ✓ | – |
| VLCDoC My work: VLCDoC: Vision-Language Contrastive Pre-Training Model for Cross-Modal Document Classification | 2023 | Cross-modal | VL contrastive | – | ✓ | – | – |
| Pix2Struct | 2023 | Vision→Text | Seq2seq | Implicit | ✓ | ✓ | – |
| UDOP | 2023 | Vision+Text+Layout | Generative | ✓ | ✓ | Partial | – |
| DocLLM / LayoutLLM | 2024 | Vision+Text+Layout | Instruction tuning | ✓ | ✓ | Partial | – |
| GlobalDoc My work: GlobalDoc: A Cross-Modal Vision-Language Framework for Real-World Document Image Retrieval and Classification | 2025 | Cross-modal | Unified VL | ✓ | ✓ | – | – |
13 representation models for documents, BERT (2019) to GlobalDoc (2025). Rows marked My work are my models and link to their publication pages. Source: Table I of the paper.
| Paradigm | Multimodal | Doc structure | Persistent | Temporal | Provenance | Governance |
|---|---|---|---|---|---|---|
| Continual learning | – | – | ✓ | – | – | – |
| Agent memory | Partial | – | ✓ | Partial | Partial | – |
| Multimodal RAG | ✓ | ✓ | Ext. index | – | Page | – |
| Commercial LLM memory | Partial | – | Context only | – | – | – |
| Target 𝒞* | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
4 adjacent paradigms against the conjunctive target 𝒞*, which needs all 6 properties jointly. Qualified marks such as Ext. index or Page are the paper’s own. Source: Table II of the paper.
§ III-A–C Formalization
The state, and what happens to it
Persistent state at session t is a six-part tuple St = (Et, Kt, Γt, 𝒱t, Pt, At). Algorithm 1 moves a document through observe, corroborate, validate, write, consolidate and reconcile, with forgetting on request. Step through it on a fictional contract.
St = (Et, Kt, Γt, 𝒱t, Pt, At)
- Et Episodic evidence buffer
- Admitted observations, appended with a timestamp and their provenance.
- Kt Consolidated semantic store
- Schemas and facts abstracted from episodic traces.
- Γt Associative relation graph
- Relations between entities, documents and schemas.
- 𝒱t Temporal validity index
- When each claim holds; closed when newer evidence supersedes it.
- Pt Provenance ledger
- Links every claim back to the document, region and version it came from.
- At Authorization and policy state
- Who may read or change what; updated only by policy events.
Invariant. Provenance is an invariant across every operation.
Algorithm 1 · State transition, one session
-
Observe Fig. 3 · Acquire
Read the incoming document Dt as (V, T, B, G, H, M).
Reads: — Writes: —
-
Corroborate
Check the observation against prior provenance.
Reads: Kt Pt Writes: —
-
Validate Fig. 3 · Validate
Pre-write gate: admit the evidence or leave the state unchanged.
Reads: — Writes: —
If the gate rejects, St+1 = St and the algorithm skips to Answer.
-
Write Fig. 3 · Write
Append the admitted evidence to the episodic store and extend provenance.
Reads: — Writes: Et Pt
-
Consolidate Fig. 3 · Consolidate
Abstract episodic traces into schemas and relations.
Reads: — Writes: Kt Γt
-
Reconcile Fig. 3 · Update & reconcile
Check against temporal validity and apply supersession.
Reads: — Writes: Kt 𝒱t Pt
-
Keep policy
Authorization stays as is unless a policy event occurs.
Reads: At Writes: —
-
Forget (on request) Fig. 3 · Forget
If a deletion request arrives, run unlearning across every store.
Reads: — Writes: Et Kt Γt 𝒱t Pt
-
Answer
Answer the query from the updated state, citing provenance.
Reads: Et Kt Γt 𝒱t Pt At Writes: —
§ III-B Temporal supersession
Five ways new evidence meets old
The supersession operator 𝒮sup resolves each conflict with one of five outcomes. Read four documents in order and watch the outcomes accumulate.
The five outcomes, as a table
| Outcome | What happens | Example from the demo stream |
|---|---|---|
| Support | New evidence agrees with the stored claim and corroborates it. Named in the paper; description is our reading. | Liability cap AMD-1, “unchanged”Corroborates §7.1 of the MSA. |
| Supersede | The earlier claim's validity interval is closed at the effective time; the new evidence is authoritative from then on. Defined in the paper. | Payment terms AMD-1, “45 days”Closes “30 days” on 2026-10-31; “45 days” is valid from 2026-11-01. |
| Scope-coexist | Claims with different populations or authority both stand, e.g. a preliminary statement next to its audited version. Defined in the paper. | Liability cap SOW-2, “€400,000 / year”Different scope from the MSA cap: both stand. |
| Retain conflict | The contradiction cannot be resolved yet, so both claims are kept and marked as conflicting. Named in the paper; description is our reading. | Express pallet rate INV-2026-0117, “€68.00”Contract rate since 2026-11-01 is €71.50; the line is €42.00 below it. Both claims are kept and flagged. |
| Escalate for review | The conflict is handed to a person before the state changes. Named in the paper; description is our reading. | Payment terms INV-2026-0117, “30 days”Contradicts the 45 days in force since 2026-11-01; routed to a person before anything is paid. |
§ III-D Formal guarantees
Remembering wrong things: the contamination bound
A persistent system can carry one error into every future session. Theorem III.4 bounds the probability that state is contaminated after k updates; Corollary III.7 says the gate sets the rate, never the asymptote.
Contamination after k updates
Pcont(k) = 1 − (1 − pe · pc)k
Expected Memory Harm
E[MH(k)] = wm · Pcont(k) · DKL(Ptrue ‖ Pfalse)
Safe horizon · Corollary III.7
k ≤ ln(1 − δ) / ln(1 − pe · pc)
Variables · assumes step independence
- pe
- observation error rate
- pc
- gate false-accept rate
- k
- number of state updates (sessions)
- δ
- tolerated contamination probability
Below, q = pe · pc is the chance that a single update admits an erroneous observation, so Pcont(k) = 1 − (1 − q)k.
Worked example · illustrative values pe = 0.05, pc = 0.1, δ = 0.05
- Per-update contamination
- 0.005
- q = pe · pc = 0.05 · 0.1
- Contamination after 100 updates
- 39.42%
- Pcont(100) = 1 − (1 − 0.005)100
- Safe horizon for δ = 0.05
- k* = 10.23
- ln(1 − 0.05) / ln(1 − 0.005): the first 10 updates stay within δ.
Values of Pcont(k)
| Updates k | Pcont(k) |
|---|---|
| 1 | 0.50% |
| 10 | 4.89% |
| 100 | 39.42% |
| 1,000 | 99.33% |
| 10,000 | ≈ 100% |
Corollary III.7
- A perfect gate (pc = 0) gives zero contamination.
- For any pc > 0, contamination approaches 1 as k grows: the gate controls the rate, not the asymptote.
Conjecture III.8
schema repair suffices for zero leakage under three stated conditions, which the authors note do not generally hold.
§ III-E Substrate composition
No single store does it all
Table III rates six substrates on six capabilities. Combine them to see which capabilities a composition covers natively.
| Substrate | Persist | Inspect | Relate | Generalize | Delete | Ground |
|---|---|---|---|---|---|---|
| Vector store | ✓ | ✓ | Partial | Partial | ✓ | Partial |
| Knowledge graph | ✓ | ✓ | ✓ | Partial | ✓ | ✓ |
| Document database | ✓ | ✓ | ✓ | – | ✓ | ✓ |
| Parametric update | ✓ | – | Implicit | ✓ | – | – |
| Episodic memory | ✓ | ✓ | Partial | – | ✓ | ✓ |
| Hybrid | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
- ✓
- native
- Partial
- achievable with extra engineering
- Implicit
- emergent only
- –
- not achievable by design
The paper deliberately does not pick a substrate: the specification constrains behavior, not architecture. Source: Table III of the paper.
§ IV-A The statefulness audit
Ten benchmarks, eight criteria, one blind spot
Table IV codes ten representative document benchmarks against eight statefulness criteria. None of the ten document benchmarks tests cross-session state evolution.
| Benchmark | Sequential | Cross-doc | Family | Temporal | Provenance | Forgetting | Session | Answerability |
|---|---|---|---|---|---|---|---|---|
| DocVQA | – | – | – | – | Partial | – | – | – |
| MP-DocVQA | – | – | – | – | Page | – | – | – |
| DUDE | – | – | – | – | Page | – | – | Yes |
| LongDocURL | – | – | – | – | Box | – | – | – |
| M-LongDoc | – | Partial | – | – | Partial | – | – | – |
| ViDoRe | – | – | – | – | Page | – | Ext. index | – |
| ChartQA | – | – | – | – | Element | – | – | – |
| FUNSD | – | – | – | – | Box | – | – | – |
| PubTables-1M | – | – | – | – | Cell | – | – | – |
| QASPER | – | – | – | – | Section | – | – | Yes |
| Adjacent LoCoMo / SYNAPSE is adjacent, text-only conversational evaluation. | ||||||||
| LoCoMo / SYNAPSE | Yes | – | Partial | Partial | Text-only | – | Yes | Yes |
| Specification The proposed-harness row is a specification, not a measured result. | ||||||||
| Proposed harness | Yes | Yes | Yes | Yes | Span | Yes | Yes | Yes |
Finding None of the ten document benchmarks tests cross-session state evolution.
- Yes
- the protocol requires the capability, with ground-truth labels
- Partial
- a neighboring capability without the full persistent-state criterion
- –
- neither required nor measured
- Ext. index
- retrieval over an external corpus, without session-level persistence
Criteria: Sequential: Sequential presentation; Cross-doc: Cross-document dependency; Family: Repeated-family benefit; Temporal: Temporal revision; Provenance: Provenance persistence; Forgetting: Verifiable forgetting; Session: Session persistence; Answerability: Answerability.
The Provenance column records grounding granularity (page, box, cell, …) rather than Yes / –. Source: Table IV of the paper.
§ IV-B–D Longitudinal benchmark
Measuring memory with counterfactuals
The decisive experiment: same base model, same future documents, different prior histories. Three arms and five metrics characterize the benefit, cost, risk and governability of state.
Hypothetical inputs. The paper specifies this harness; it reports no measurements.
Stream
- D1
- D2
- Dk
One document per session, before the test.
Skclean
Clean twin for MH
The warm state the same documents would give with no corrupted observations.
-
Sk
Warm state
Carries consolidated state from documents D1 … Dk.
-
Rk
Retrieval-only
An append-only index of the same documents, with no consolidation.
-
S0 = ∅
Cold baseline
Starts from empty state on every task.
Every arm is tested on
Dk+1
The same next document; quality Q and cost C are compared.
Deletion audit for FF and CR
St
State before the deletion request
S′t
State after forgetting Xdel
Success criterion
Positive Experience Gain and Cost Efficiency, bounded Memory Harm, and high Forgetting Fidelity and Coverage Retention, under all eight mandatory properties.
The five metrics
| Metric | Definition | Measures | Target |
|---|---|---|---|
| EG Experience Gain | {"id"=>"eg", "abbr"=>"EG", "name"=>"Experience Gain", "equation"=>"EG(k) = Q(D_k+1 | S_k) − Q(D_k+1 | R_k)", "measures"=>"Benefit of consolidation beyond retrieval access.", "good"=>"positive"} | Benefit of consolidation beyond retrieval access. | Positive |
| CE Cost Efficiency | {"id"=>"ce", "abbr"=>"CE", "name"=>"Cost Efficiency", "equation"=>"CE(k) = 1 − C(D_k+1 | S_k) / C(D_k+1 | S_0)", "measures"=>"Inference cost saved by warm state, in tokens or FLOPs.", "good"=>"positive"} | Inference cost saved by warm state, in tokens or FLOPs. | Positive |
| MH Memory Harm | {"id"=>"mh", "abbr"=>"MH", "name"=>"Memory Harm", "equation"=>"MH(k) = Q(D_k+1 | S_k^clean) − Q(D_k+1 | S_k)", "measures"=>"Degradation induced by state; positive values mean memory hurt.", "good"=>"bounded"} | Degradation induced by state; positive values mean memory hurt. | Bounded |
| FF Forgetting Fidelity | {"id"=>"ff", "abbr"=>"FF", "name"=>"Forgetting Fidelity", "equation"=>"FF = 1 − Leak(X_del | S′_t) / Leak(X_del | S_t)", "measures"=>"How much of a deleted target still leaks from the post-deletion state (probe-based mutual-information bound).", "good"=>"high"} | How much of a deleted target still leaks from the post-deletion state (probe-based mutual-information bound). | High |
| CR Coverage Retention | {"id"=>"cr", "abbr"=>"CR", "name"=>"Coverage Retention", "equation"=>"CR = exp( Q(Y_ret | S′_t) − Q(Y_ret | S_t) )", "measures"=>"Whether knowledge that should survive a deletion still does.", "good"=>"high"} | Whether knowledge that should survive a deletion still does. | High |
Memory Harm protocol. Run a clean stream and a corrupted one over the same documents; the corrupted stream replaces a fraction ρc of observations with plausible but wrong values, such as a misread tax ID or a transposed table cell.
Cumulative cost. CEcum(k) = 1 − Σ warm costs / Σ cold costs over sessions 1 … k+1. A memory system pays for itself when CEcum(k) > 0 for the target stream length, even if per-task CE is zero in some sessions.
Analysis. A mixed-effects model with a log-exposure term f(k) = log(1 + k), a warm–cold indicator, family-level random intercepts and slopes, and AR(1) residuals; a paired bootstrap over streams for small samples.
Untick what a benchmark lacks: the success criterion only holds under all eight.
- Sequential presentation Documents arrive across sessions, and state persists between them.
- Cross-document dependency Some questions need evidence seen only in earlier sessions.
- Evolving content Controlled revisions, amendments and supersessions.
- Consolidation opportunities Repeated instances of a document family.
- Forgetting requirements Deletion requests, checked for an unrecoverable target and intact unrelated knowledge.
- Provenance queries Conclusions must be traced back to their source regions.
- Session boundaries Context is cleared, so only the persistent store survives.
- Abstention Unanswerable queries the system must decline rather than answer from stale or contaminated state.
§ V-C Governance
Forgetting, verifiably
Deleting the file is level one of five. Passing one level does not imply success at the next. Current Document AI implements Levels 1–2; skipping Level 3 leaves non-zero leakage in the semantic store.
-
Level 1
Physical object removal
Delete the raw file.
-
Level 2
Representation invalidation
Remove embeddings and graph nodes derived from it.
-
Level 3
Consolidated schema repair
Re-consolidate schemas that depended on the deleted evidence and repair links.
-
Level 4
Parametric unlearning
Remove facts absorbed into model weights: exact retraining, influence-based approximation, certified residual checks.
-
Level 5
Certified non-recoverability
Audit residual leakage cryptographically or statistically, then issue a certificate.
Passing one level does not imply success at the next. Current Document AI implements Levels 1–2; skipping Level 3 leaves non-zero leakage in the semantic store.
§ V-B, V-D Deployment
Where it matters, and what it must survive
Schemas abstract over evidence without cutting the chain back to the document, location and version. A system that cannot cite the exact table cell and footnote behind a claim cannot be deployed in an audit.
Domains
- Legal discovery
- Millions of documents and non-monotonic amendments.
- Healthcare
- Longitudinal patient models.
- Financial auditing
- Unbroken provenance chains.
- Scientific literature
- Synthesis over superseded findings.
Constraints
- Throughput and efficiency at scale
- Consistency, availability and cost trade-offs
- Multi-tenant isolation
- Permission-aware governance within a single document
- Security against persistent poisoning
- Retention and data-residency rules for remembered information
σ₂ ↝ σ₁ ↝ {e₁, e₂, e₃} ↝ {Di, ℓi, vi}
Schemas σ abstract over evidence e, which points back to document Di, location ℓi and version vi. Bitemporal state tracks publication time, event time, effective time, ingestion time.
§ V-E Research agenda
Six open problems
An implementation-neutral agenda: each item is a concrete research question, not a product feature.
-
Problem 1
Provenance-preserving consolidation
Abstract visual observations into schemas without losing links to source bounding boxes.
-
Problem 2
Adaptive evidence acquisition
Stop inspecting pages when the expected information gain falls below the compute cost.
-
Problem 3
Memory-conditioned top-down parsing
Show that warm-state parsers speed up layout detection on recurring document families.
-
Problem 4
Bitemporal state and non-monotonic supersession
Model publication, event, effective and ingestion time without losing audit trails.
-
Problem 5
Certified machine unlearning
Repair knowledge graphs and parametric priors after erasure requests.
-
Problem 6
Policy-aware governed memory
Embed multi-tenant access control in late-interaction retrieval and in the state itself.
§ V-F Limitations
What the paper does not claim
The paper is a survey and specification: it reports formal results and an audit, not experimental measurements.
The survey leans towards English, Latin-script, modern documents; low-resource languages and historical manuscripts are under-represented.
Cognitive science motivates hypotheses, not optimal design; the cortical-consolidation mapping is an analogy.
The state tuple, lifecycle and harness are formal specifications, not validated implementations.
Obtaining genuinely novel document streams, absent from training data, is the main benchmark limitation.
“The field has learned to read documents. It must now learn to remember them.”
Cite
@article{bakkali2026amnesia,
title = {Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligence},
author = {Bakkali, Souhail and Merimi, Ayoub},
journal = {arXiv preprint arXiv:2609.32041},
year = {2026},
url = {https://arxiv.org/abs/2609.32041}
}
Figures on this page are transcribed from the arXiv HTML version. Interactive examples use fictional documents and are marked “Illustrative”.