arXiv:2609.32041 · cs.CV · 25 September 2026

Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligence

Souhail BakkaliAyoub Merimi

Survey with a formal framework, a benchmark audit and an evaluation harness. Document AI reads, then forgets; this survey argues the fix is not more scale, context or retrieval, but persistent, provenance-linked state, and specifies how to build and measure it. Every figure below is interactive.

6state components
6lifecycle operations
10benchmarks audited
8statefulness criteria
5counterfactual metrics
5unlearning levels

§ I Introduction

The statelessness bottleneck

Document systems are modeled as single-shot functions, ŷ = argmax P(y | D; θ): frozen parameters, no session index. Whatever one document taught is gone when the next one arrives.

  1. Document AI reads, then forgets

    Systems extract fields, reason over tables and ground answers to page regions, yet process each document as if it were the first. An amendment read the next day starts from scratch: no schema retained, no contradiction detected.

  2. It is a design choice, not a scale failure

    Current systems are stateless functions, y = argmax P(y | D; θ), with frozen parameters and no session index. The paper calls this the statelessness bottleneck.

  3. Scaling, context and retrieval do not remove it

    Storage provides persistence and retrieval provides access, but neither consolidates observations into knowledge that improves how the next document is processed.

  4. So the paper specifies persistent, evidence-grounded state

    It formalizes the state, its lifecycle and invariants, bounds the risk of remembering wrong things, audits ten benchmarks, and proposes a longitudinal harness with five counterfactual metrics.

Document tuple · Eq. 3

D = (V, T, B, G, H, M)

V
Visual tensors
T
Text tokens
B
Bounding boxes
G
Structural relation graphs
H
Hierarchical parse trees
M
Metadata
Conjunctive target 𝒞* · Eq. 2

All six properties are required jointly; no adjacent paradigm has them all.

  • Multimodal evidence
  • Document structure
  • Persistence
  • Temporal validity
  • Provenance
  • Governance

§ II Capability evolution

Where the field stands: an empty top row

Fig. 2 places Document AI lineages by how much they persist. Cross-modal models stop at the page with no persistence; retrieval reaches an external store; text-only agent memory reaches episodic state. Nothing reaches semantic or governed state.

Capability map · Fig. 2 InteractiveFrom the paper

Target, spanning the Governed and Semantic bands: 𝒞*

  • Multimodal evidence
  • Document structure
  • Persistence
  • Temporal validity
  • Provenance
  • Governance
  1. GovernedEmpty: no surveyed system reaches this band
  2. SemanticEmpty: no surveyed system reaches this band
  3. Episodic
    • Text-only agent memory , scope: Conversation / session

      • MemGPT
      • LoCoMo
      • SYNAPSE
      • GAAMA
  4. External store
    • Retrieval systems , scope: Collection

      • ColPali / ViDoRe
      • VisRAG
      • GraphRAG
      • LightRAG
  5. No persistence
    • Cross-modal & layout models, single-page benchmarks , scope: Page

      • DocVQA
      • ChartQA
      • FUNSD
      • LayoutLM
      • Donut
      • EAML
      • SelfDoc
      • UDoc
      • VLCDoC
      • GlobalDoc
    • Multi-page benchmarks , scope: Multi-page document

      • MP-DocVQA
      • DUDE
      • LongDocURL
      • M-LongDoc
The Semantic and Governed bands are empty: no surveyed system reaches them. The target 𝒞* sits there. Redrawn from Fig. 2 of the paper, which places groups of systems, not individual cells, by how much they persist; position within a band carries no meaning.

§ II-B Representation lineage

A decade of representations, no native state

Tables I and II: thirteen representation models and four adjacent paradigms, against the properties the target needs.

Table I · Representation lineage InteractiveFrom the paper
Table I: 13 representation models for documents, with their modality, pretraining and four properties.
Method Year Modality Pretraining Layout Visual OCR-free Native state
BERT 2019 Text MLM – – – –
LayoutLMv1–v3 2020–22 Text+Layout+Vision MLM/MIM ✓ ✓ – –
EAML My work: EAML: Ensemble Self-Attention-Based Mutual Learning Network for Document Image Classification 2021 Image+Text Mutual learning – ✓ – –
SelfDoc 2021 Cross-modal Self-supervised ✓ ✓ – –
UDoc 2021 Cross-modal Unified alignment ✓ ✓ – –
DiT 2022 Vision MIM Implicit ✓ ✓ –
Bi-VLDoc 2022 Cross-modal Bidirectional VL ✓ ✓ – –
Donut 2022 Vision→Text Seq2seq Implicit ✓ ✓ –
VLCDoC My work: VLCDoC: Vision-Language Contrastive Pre-Training Model for Cross-Modal Document Classification 2023 Cross-modal VL contrastive – ✓ – –
Pix2Struct 2023 Vision→Text Seq2seq Implicit ✓ ✓ –
UDOP 2023 Vision+Text+Layout Generative ✓ ✓ Partial –
DocLLM / LayoutLLM 2024 Vision+Text+Layout Instruction tuning ✓ ✓ Partial –
GlobalDoc My work: GlobalDoc: A Cross-Modal Vision-Language Framework for Real-World Document Image Retrieval and Classification 2025 Cross-modal Unified VL ✓ ✓ – –

13 representation models for documents, BERT (2019) to GlobalDoc (2025). Rows marked My work are my models and link to their publication pages. Source: Table I of the paper.

Table II · Adjacent paradigms against 𝒞* InteractiveFrom the paper
Table II: 4 adjacent paradigms and the target 𝒞*, against 6 properties.
Paradigm Multimodal Doc structure Persistent Temporal Provenance Governance
Continual learning – – ✓ – – –
Agent memory Partial – ✓ Partial Partial –
Multimodal RAG ✓ ✓ Ext. index – Page –
Commercial LLM memory Partial – Context only – – –
Target 𝒞* ✓ ✓ ✓ ✓ ✓ ✓

4 adjacent paradigms against the conjunctive target 𝒞*, which needs all 6 properties jointly. Qualified marks such as Ext. index or Page are the paper’s own. Source: Table II of the paper.

§ III-A–C Formalization

The state, and what happens to it

Persistent state at session t is a six-part tuple St = (Et, Kt, Γt, 𝒱t, Pt, At). Algorithm 1 moves a document through observe, corroborate, validate, write, consolidate and reconcile, with forgetting on request. Step through it on a fictional contract.

State lifecycle · Eq. 4 · Fig. 3 · Algorithm 1 Interactive Illustrative

St = (Et, Kt, Γt, 𝒱t, Pt, At)

Et Episodic evidence buffer
Admitted observations, appended with a timestamp and their provenance.
Kt Consolidated semantic store
Schemas and facts abstracted from episodic traces.
Γt Associative relation graph
Relations between entities, documents and schemas.
𝒱t Temporal validity index
When each claim holds; closed when newer evidence supersedes it.
Pt Provenance ledger
Links every claim back to the document, region and version it came from.
At Authorization and policy state
Who may read or change what; updated only by policy events.

Invariant. Provenance is an invariant across every operation.

Algorithm 1 · State transition, one session

  1. Observe Fig. 3 · Acquire

    Read the incoming document Dt as (V, T, B, G, H, M).

    Reads: — Writes: —

  2. Corroborate

    Check the observation against prior provenance.

    Reads: Kt Pt Writes: —

  3. Validate Fig. 3 · Validate

    Pre-write gate: admit the evidence or leave the state unchanged.

    Reads: — Writes: —

    If the gate rejects, St+1 = St and the algorithm skips to Answer.

  4. Write Fig. 3 · Write

    Append the admitted evidence to the episodic store and extend provenance.

    Reads: — Writes: Et Pt

  5. Consolidate Fig. 3 · Consolidate

    Abstract episodic traces into schemas and relations.

    Reads: — Writes: Kt Γt

  6. Reconcile Fig. 3 · Update & reconcile

    Check against temporal validity and apply supersession.

    Reads: — Writes: Kt 𝒱t Pt

  7. Keep policy

    Authorization stays as is unless a policy event occurs.

    Reads: At Writes: —

  8. Forget (on request) Fig. 3 · Forget

    If a deletion request arrives, run unlearning across every store.

    Reads: — Writes: Et Kt Γt 𝒱t Pt

  9. Answer

    Answer the query from the updated state, citing provenance.

    Reads: Et Kt Γt 𝒱t Pt At Writes: —

The six components are Eq. 4 of the paper; the steps, their order and the components each one reads or writes follow Algorithm 1 and the lifecycle of Fig. 3. Register contents are our illustration on fictional documents: a dashed outline means the step reads a component, a tint means it writes it. Fictional documents. The demo runs entirely in your browser; it illustrates the ideas and is not production software.

§ III-B Temporal supersession

Five ways new evidence meets old

The supersession operator 𝒮sup resolves each conflict with one of five outcomes. Read four documents in order and watch the outcomes accumulate.

Temporal supersession · the five outcomes of 𝒮sup Interactive Illustrative
The five outcomes, as a table
OutcomeWhat happensExample from the demo stream
Support New evidence agrees with the stored claim and corroborates it. Named in the paper; description is our reading. Liability cap AMD-1, “unchanged”Corroborates §7.1 of the MSA.
Supersede The earlier claim's validity interval is closed at the effective time; the new evidence is authoritative from then on. Defined in the paper. Payment terms AMD-1, “45 days”Closes “30 days” on 2026-10-31; “45 days” is valid from 2026-11-01.
Scope-coexist Claims with different populations or authority both stand, e.g. a preliminary statement next to its audited version. Defined in the paper. Liability cap SOW-2, “€400,000 / year”Different scope from the MSA cap: both stand.
Retain conflict The contradiction cannot be resolved yet, so both claims are kept and marked as conflicting. Named in the paper; description is our reading. Express pallet rate INV-2026-0117, “€68.00”Contract rate since 2026-11-01 is €71.50; the line is €42.00 below it. Both claims are kept and flagged.
Escalate for review The conflict is handed to a person before the state changes. Named in the paper; description is our reading. Payment terms INV-2026-0117, “30 days”Contradicts the 45 days in force since 2026-11-01; routed to a person before anything is paid.
How to read it. Read the four fictional documents in order: each field meets the claims already stored and gets one outcome; the ledger is the resulting validity index. Outcome names come from the paper, which defines supersede and scope-coexist and only names support, retain conflict, escalate for review; for those, the descriptions are our reading. “New claim” is not one of the five: nothing was stored yet. Fictional documents. The demo runs entirely in your browser; it illustrates the ideas and is not production software.

§ III-D Formal guarantees

Remembering wrong things: the contamination bound

A persistent system can carry one error into every future session. Theorem III.4 bounds the probability that state is contaminated after k updates; Corollary III.7 says the gate sets the rate, never the asymptote.

Theorem III.4 · Contamination bound Interactive Illustrative values

Contamination after k updates

Pcont(k) = 1 − (1 − pe · pc)k

Expected Memory Harm

E[MH(k)] = wm · Pcont(k) · DKL(Ptrue ‖ Pfalse)

Safe horizon · Corollary III.7

k ≤ ln(1 − δ) / ln(1 − pe · pc)

Variables · assumes step independence

pe
observation error rate
pc
gate false-accept rate
k
number of state updates (sessions)
δ
tolerated contamination probability

Below, q = pe · pc is the chance that a single update admits an erroneous observation, so Pcont(k) = 1 − (1 − q)k.

Worked example · illustrative values pe = 0.05, pc = 0.1, δ = 0.05

Per-update contamination
0.005
q = pe · pc = 0.05 · 0.1
Contamination after 100 updates
39.42%
Pcont(100) = 1 − (1 − 0.005)100
Safe horizon for δ = 0.05
k* = 10.23
ln(1 − 0.05) / ln(1 − 0.005): the first 10 updates stay within δ.
Values of Pcont(k)
Worked example: q = 0.005. Illustrative values, not measurements.
Updates kPcont(k)
10.50%
104.89%
10039.42%
1,00099.33%
10,000≈ 100%

Corollary III.7

  • A perfect gate (pc = 0) gives zero contamination.
  • For any pc > 0, contamination approaches 1 as k grows: the gate controls the rate, not the asymptote.

Conjecture III.8

schema repair suffices for zero leakage under three stated conditions, which the authors note do not generally hold.

Move the sliders or pick a gate. The chart plots Pcont(k) with faint curves for a gate ten times weaker and ten times stronger; the dashed line is δ and the marker is the horizon k*. Equations from Theorem III.4 and Corollary III.7 of the paper; every number here is computed from illustrative inputs, not measured.

§ III-E Substrate composition

No single store does it all

Table III rates six substrates on six capabilities. Combine them to see which capabilities a composition covers natively.

Table III · Substrate options InteractiveIllustrativeFrom the paper
Table III: single substrates and a hybrid, rated on 6 capabilities.
Substrate Persist Inspect Relate Generalize Delete Ground
Vector store ✓ ✓ Partial Partial ✓ Partial
Knowledge graph ✓ ✓ ✓ Partial ✓ ✓
Document database ✓ ✓ ✓ – ✓ ✓
Parametric update ✓ – Implicit ✓ – –
Episodic memory ✓ ✓ Partial – ✓ ✓
Hybrid ✓ ✓ ✓ ✓ ✓ ✓
✓
native
Partial
achievable with extra engineering
Implicit
emergent only
–
not achievable by design

The paper deliberately does not pick a substrate: the specification constrains behavior, not architecture. Source: Table III of the paper.

§ IV-A The statefulness audit

Ten benchmarks, eight criteria, one blind spot

Table IV codes ten representative document benchmarks against eight statefulness criteria. None of the ten document benchmarks tests cross-session state evolution.

Table IV · Statefulness audit InteractiveFrom the paper
Table IV: 10 document benchmarks, an adjacent benchmark and the proposed harness, coded against 8 statefulness criteria.
Benchmark Sequential Cross-doc Family Temporal Provenance Forgetting Session Answerability
DocVQA – – – – Partial – – –
MP-DocVQA – – – – Page – – –
DUDE – – – – Page – – Yes
LongDocURL – – – – Box – – –
M-LongDoc – Partial – – Partial – – –
ViDoRe – – – – Page – Ext. index –
ChartQA – – – – Element – – –
FUNSD – – – – Box – – –
PubTables-1M – – – – Cell – – –
QASPER – – – – Section – – Yes
Adjacent LoCoMo / SYNAPSE is adjacent, text-only conversational evaluation.
LoCoMo / SYNAPSE Yes – Partial Partial Text-only – Yes Yes
Specification The proposed-harness row is a specification, not a measured result.
Proposed harness Yes Yes Yes Yes Span Yes Yes Yes

Finding None of the ten document benchmarks tests cross-session state evolution.

Yes
the protocol requires the capability, with ground-truth labels
Partial
a neighboring capability without the full persistent-state criterion
–
neither required nor measured
Ext. index
retrieval over an external corpus, without session-level persistence

Criteria: Sequential: Sequential presentation; Cross-doc: Cross-document dependency; Family: Repeated-family benefit; Temporal: Temporal revision; Provenance: Provenance persistence; Forgetting: Verifiable forgetting; Session: Session persistence; Answerability: Answerability.

The Provenance column records grounding granularity (page, box, cell, …) rather than Yes / –. Source: Table IV of the paper.

§ IV-B–D Longitudinal benchmark

Measuring memory with counterfactuals

The decisive experiment: same base model, same future documents, different prior histories. Three arms and five metrics characterize the benefit, cost, risk and governability of state.

Counterfactual harness · after Fig. 4 Interactive Hypothetical inputs

Hypothetical inputs. The paper specifies this harness; it reports no measurements.

Stream

  1. D1
  2. D2
  3. Dk

One document per session, before the test.

Skclean

Clean twin for MH

The warm state the same documents would give with no corrupted observations.

  1. Sk

    Warm state

    Carries consolidated state from documents D1 … Dk.

  2. Rk

    Retrieval-only

    An append-only index of the same documents, with no consolidation.

  3. S0 = ∅

    Cold baseline

    Starts from empty state on every task.

Every arm is tested on

Dk+1

The same next document; quality Q and cost C are compared.

Deletion audit for FF and CR

St

State before the deletion request

S′t

State after forgetting Xdel

Success criterion

Positive Experience Gain and Cost Efficiency, bounded Memory Harm, and high Forgetting Fidelity and Coverage Retention, under all eight mandatory properties.

The five metrics
MetricDefinitionMeasuresTarget
EG Experience Gain{"id"=>"eg", "abbr"=>"EG", "name"=>"Experience Gain", "equation"=>"EG(k) = Q(D_k+1 | S_k) − Q(D_k+1 | R_k)", "measures"=>"Benefit of consolidation beyond retrieval access.", "good"=>"positive"}Benefit of consolidation beyond retrieval access.Positive
CE Cost Efficiency{"id"=>"ce", "abbr"=>"CE", "name"=>"Cost Efficiency", "equation"=>"CE(k) = 1 − C(D_k+1 | S_k) / C(D_k+1 | S_0)", "measures"=>"Inference cost saved by warm state, in tokens or FLOPs.", "good"=>"positive"}Inference cost saved by warm state, in tokens or FLOPs.Positive
MH Memory Harm{"id"=>"mh", "abbr"=>"MH", "name"=>"Memory Harm", "equation"=>"MH(k) = Q(D_k+1 | S_k^clean) − Q(D_k+1 | S_k)", "measures"=>"Degradation induced by state; positive values mean memory hurt.", "good"=>"bounded"}Degradation induced by state; positive values mean memory hurt.Bounded
FF Forgetting Fidelity{"id"=>"ff", "abbr"=>"FF", "name"=>"Forgetting Fidelity", "equation"=>"FF = 1 − Leak(X_del | S′_t) / Leak(X_del | S_t)", "measures"=>"How much of a deleted target still leaks from the post-deletion state (probe-based mutual-information bound).", "good"=>"high"}How much of a deleted target still leaks from the post-deletion state (probe-based mutual-information bound).High
CR Coverage Retention{"id"=>"cr", "abbr"=>"CR", "name"=>"Coverage Retention", "equation"=>"CR = exp( Q(Y_ret | S′_t) − Q(Y_ret | S_t) )", "measures"=>"Whether knowledge that should survive a deletion still does.", "good"=>"high"}Whether knowledge that should survive a deletion still does.High

Memory Harm protocol. Run a clean stream and a corrupted one over the same documents; the corrupted stream replaces a fraction ρc of observations with plausible but wrong values, such as a misread tax ID or a transposed table cell.

Cumulative cost. CEcum(k) = 1 − Σ warm costs / Σ cold costs over sessions 1 … k+1. A memory system pays for itself when CEcum(k) > 0 for the target stream length, even if per-task CE is zero in some sessions.

Analysis. A mixed-effects model with a log-exposure term f(k) = log(1 + k), a warm–cold indicator, family-level random intercepts and slopes, and AR(1) residuals; a paired bootstrap over streams for small samples.

Pick a metric to see which arms it compares, then move the hypothetical inputs or try a scenario. Adapted from Fig. 4 of the paper, with the retrieval-only arm described in its text. Arms, metrics and the criterion are the paper's; the paper reports no results for them.
Required design · Eight mandatory properties Interactive

Untick what a benchmark lacks: the success criterion only holds under all eight.

  1. Sequential presentation Documents arrive across sessions, and state persists between them.
  2. Cross-document dependency Some questions need evidence seen only in earlier sessions.
  3. Evolving content Controlled revisions, amendments and supersessions.
  4. Consolidation opportunities Repeated instances of a document family.
  5. Forgetting requirements Deletion requests, checked for an unrecoverable target and intact unrelated knowledge.
  6. Provenance queries Conclusions must be traced back to their source regions.
  7. Session boundaries Context is cleared, so only the persistent store survives.
  8. Abstention Unanswerable queries the system must decline rather than answer from stale or contaminated state.
Property names from the paper; the one-line glosses are ours.

§ V-C Governance

Forgetting, verifiably

Deleting the file is level one of five. Passing one level does not imply success at the next. Current Document AI implements Levels 1–2; skipping Level 3 leaves non-zero leakage in the semantic store.

Verifiable forgetting · Fig. 6 · Algorithm 2 Interactive Illustrative
  1. Level 1

    Physical object removal

    Delete the raw file.

    Implemented in current Document AI Evaluated by: Forgetting Fidelity · Coverage Retention

  2. Level 2

    Representation invalidation

    Remove embeddings and graph nodes derived from it.

    Implemented in current Document AI Evaluated by: Forgetting Fidelity · Coverage Retention

  3. Level 3

    Consolidated schema repair

    Re-consolidate schemas that depended on the deleted evidence and repair links.

    Unbuilt Evaluated by: Forgetting Fidelity · Coverage Retention

  4. Level 4

    Parametric unlearning

    Remove facts absorbed into model weights: exact retraining, influence-based approximation, certified residual checks.

    Unbuilt Evaluated by: Additional certification

  5. Level 5

    Certified non-recoverability

    Audit residual leakage cryptographically or statistically, then issue a certificate.

    Unbuilt Evaluated by: Additional certification

Passing one level does not imply success at the next. Current Document AI implements Levels 1–2; skipping Level 3 leaves non-zero leakage in the semantic store.

Levels, their status and how each is evaluated are Fig. 6 of the paper. The erasure run is our illustration on fictional documents; leakage is shown qualitatively (present or removed), nothing is measured. Fictional documents. The demo runs entirely in your browser; it illustrates the ideas and is not production software.

§ V-B, V-D Deployment

Where it matters, and what it must survive

Schemas abstract over evidence without cutting the chain back to the document, location and version. A system that cannot cite the exact table cell and footnote behind a claim cannot be deployed in an audit.

Domains

Legal discovery
Millions of documents and non-monotonic amendments.
Healthcare
Longitudinal patient models.
Financial auditing
Unbroken provenance chains.
Scientific literature
Synthesis over superseded findings.

Constraints

  • Throughput and efficiency at scale
  • Consistency, availability and cost trade-offs
  • Multi-tenant isolation
  • Permission-aware governance within a single document
  • Security against persistent poisoning
  • Retention and data-residency rules for remembered information
Recursive provenance · § V-B

σ₂ ↝ σ₁ ↝ {e₁, e₂, e₃} ↝ {Di, ℓi, vi}

Schemas σ abstract over evidence e, which points back to document Di, location ℓi and version vi. Bitemporal state tracks publication time, event time, effective time, ingestion time.

§ V-E Research agenda

Six open problems

An implementation-neutral agenda: each item is a concrete research question, not a product feature.

  1. Problem 1

    Provenance-preserving consolidation

    Abstract visual observations into schemas without losing links to source bounding boxes.

  2. Problem 2

    Adaptive evidence acquisition

    Stop inspecting pages when the expected information gain falls below the compute cost.

  3. Problem 3

    Memory-conditioned top-down parsing

    Show that warm-state parsers speed up layout detection on recurring document families.

  4. Problem 4

    Bitemporal state and non-monotonic supersession

    Model publication, event, effective and ingestion time without losing audit trails.

  5. Problem 5

    Certified machine unlearning

    Repair knowledge graphs and parametric priors after erasure requests.

  6. Problem 6

    Policy-aware governed memory

    Embed multi-tenant access control in late-interaction retrieval and in the state itself.

§ V-F Limitations

What the paper does not claim

The paper is a survey and specification: it reports formal results and an audit, not experimental measurements.

  • The survey leans towards English, Latin-script, modern documents; low-resource languages and historical manuscripts are under-represented.

  • Cognitive science motivates hypotheses, not optimal design; the cortical-consolidation mapping is an analogy.

  • The state tuple, lifecycle and harness are formal specifications, not validated implementations.

  • Obtaining genuinely novel document streams, absent from training data, is the main benchmark limitation.

“The field has learned to read documents. It must now learn to remember them.”

§ VI, Conclusion

Cite

@article{bakkali2026amnesia,
  title   = {Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligence},
  author  = {Bakkali, Souhail and Merimi, Ayoub},
  journal = {arXiv preprint arXiv:2609.32041},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.32041}
}

Figures on this page are transcribed from the arXiv HTML version. Interactive examples use fictional documents and are marked “Illustrative”.